The FME Evangelist

FME Evangelism - Smallworld UC Special

Contents:
FME/Smallworld Background,
FME and the Smallworld SIAS,
Behind the Scenes,
Technical Details of a Smallworld Workspace,
Data Streaming and Linking,
What’s New in the FME SpatialBiz Plug-In,
Smallworld 4.1 Support,
Training Offer.

This FME Evangelist is a special edition for the 2008 Smallworld Americas Users Conference. It details a particular scenario that we will be demonstrating at the conference: how to integrate FME Server with a web mapping tool such as SIAS (Smallworld Internet Application Server). It has good information for anyone new to FME, plus news about recent FME-Smallworld enhancements, and mentions techniques useful to most users.

Continue Reading September 27th, 2008

FME Evangelism #17: ESRI UC Special (Historical Markers, Complex Annotations, Image Server, Persistent Connections)

Contents:
1) Historical Markers
2) Complex Annotation Geometries
3) ArcGIS Image Server
4) Persistent SDE Connections
5) Text Size and Point Size
6) Quick Notes

Introduction
Hi FME’ers

In honour of the 2008 ESRI International User Conference, this post is a special on recent ESRI related updates in FME.

I think a key point of FME is how it easily integrates into spatial data workflows, simplifies processes, and helps people concentrate on using their data (rather than writing scripts or mucking about with text editors!). Therefore, for us at Safe, it’s vital that we keep our format support as up to date as possible; as even the smallest piece of unsupported functionality can force a user to take huge detours with their data.

That’s why it’s a real pleasure to be able to highlight improvements like these below. Browsing our user requests database I saw a number of funky ArcGIS functions - most of which I didn’t even realize existed - that we’d been asked to support. And here they are! Hopefully each one of these newly-implemented features will be a real time-saver.

The one caution is that, because these functions are newly added, they are very much still in beta form and subject to change and improvement as we get user feedback.

Regards,

Historical Markers
Historical markers are part of an archiving process introduced in ArcGIS v9.2 for ESRI’s SDE and Enterprise Geodatabase.

The basic idea is that - once activated - all database edits get stored in a separate archive class, with a date and time stamp. When viewing data you can - by specifying the appropriate time range - read the state of the spatial data as it existed at any particular moment in time. Additionally, an Historical Marker Manager lets you define moments in time with specific names that will obviously have greater meaning than dates (for example “As Designed” and “As Built”).

In FME2009 (as of beta build 5568) we now support querying data on the basis of these historical records. This is achieved by either selecting an historical marker on adding a reader, or entering a query into the appropriate reader parameter in an existing workspace.

Below: When adding a Geodatabase (SDE) reader to the workspace, the settings dialog includes this section. Here you can search out and specify a particular historical marker:

Below: Or you can choose to view data at a particular point in time by date:

Below: Once the reader is added to a workspace you can access historical data by setting the Archive Where Clause parameter:

Below: The Archive Where Clause is edited with this dialog. This example is a “range query” - ie the from and to dates are different.

According to the Readers and Writers Manual, a query can be a ‘moment’ query (the from and to dates are identical) or a ‘range’ query (the from and to dates are different). ESRI documentation tells me it’s also possible to query archives using attribute values (i.e. show me when <attr>=<value>) and Historical Markers, so - although I’ve no evidence - I believe this might also work in this FME parameter.

Complex Annotation Geometries
When, a few years ago, improvements were made to the FME geometry model, it enabled a whole deal of functionality which is still being unlocked. Support for complex ArcGIS annotation features is one of these.

By “complex” annotation geometries, we mean any text that is not a single piece of straight-line text, that originates at a specific point; i.e. text that is aligned to curves, arcs, polygons, multipoints, etc.

So, as of build 5555 we now support reading and writing of annotation with complex geometry.

Below: Bonus points for spotting the language of this curved annotation string.

Key things to remember:

1. You MUST have the Geometry Handling parameter (shown below) set to “Rich”. You won’t be able to write complex annotation geometries (and maybe not read) if it is set to “Classic”.
2. The data inspected in the FME Viewer will not show curved text - that’s because it isn’t supported (i.e. the Viewer is only meant for data inspection, not visualization)
3. Splines and curves will be stroked into line features; we do hope to properly support these geometries in the near future.

Above: The Geometry Handling Mode must be Rich (not Classic)

ArcGIS Image Server
As of build 5550 Safe has added a new reader to FME: ArcGIS Image Server.

ArcGIS Image Server (or as of 9.3, ArcGIS Server Image) serves up file-based raster data to other applications. There are a number of applications with interfaces to use this data, including ArcGIS, AutoCAD and MicroStation; but the data can also be delivered as a web service, and this is where the FME reader might come into the picture.

Using FME on top of ArcGIS Server Image would firstly extend the format reach, secondly open up the data to all the raster transformation tools in FME, and thirdly allow everything to be integrated with all FME ETL functionality (for example, the ability to merge in vector datasources, live feeds, other raster data, text-based data etc etc) all “on-the-fly”.

The ArcGIS Server Image format is a reader only, since the product itself doesn’t have it’s own format (we read from a service). However for writing, it (the ArcGIS Server Image product) supports as source many of the same formats FME supports as a destination, plus I strongly suspect it would also accept a service as input, so you could create an “on-the-fly” FME writer using an FME Server service as an ArcGIS source.

NB: To get access to this format requires installation of the ArcGIS Image Server Client Core.

Persistent SDE Connections
Whenever FME connects to an SDE database it carries out the require processing and then shuts down the connection to save resources. However, when multiple tasks need to be carried out, creating a new connection each time is an expensive operation.

So that’s where a new keyword for FME2009 comes in. The PERSISTENT_CONNECTION keyword specifies whether to leave database connections open once their task is complete. That way other operations can use the same connection and avoid re-connection overheads.

Below: Persistent Connections are defined via a Reader Parameter:

This keyword isn’t intended for normal standalone translations, but for when the workspace is used in “super batch” mode or on FME Server; in these cases the reported improvement in connection time is a huge 50-60%. But just remember that an open connection is, effectively, an intentional one-time memory leak, so if you are not using SDE access in all translations it may not be worth the cost in system resources.

Text Size and Point Size
A curious discussion occurred here recently about whether “text size” and “font size” were the same thing, or different. It appears (from what I understand) that they are different; text size is the size of a text feature in ground (coordinate system) units, whereas font size is the size of the font to use in “point” units.

Therefore users were disappointed to find out that setting the format attribute geodb_text_size had no effect whatsoever on the size of the output font; and because font size is unrelated to geometry, the attribute geodb_text_ref_scale had no effect either.

The solution was for us to create a new format attribute: geodb_font_size

geodb_font_size defines the size of the font used to display the text string, in point units. However, there are some important points to note:

1. Once more, you MUST have the Geometry Handling parameter set to “Rich”. The attribute would be ignored if the geometry handling mode is set to “Classic”.
2. For the sake of backwards compatibility, geodb_text_size will overrule geodb_font_size if both are set. So remove any reference to geodb_text_size if you are wanting to set a text font size.

This fix was so important for a number of users we backported it to an updated FME2008 version. For more information see the Post-CD Fixes page on fmepedia.

Quick Notes

• The “Split Multi Part Annotations” option available on the Geodatabase_MDB reader has also been exposed on the SDE Geodatabase and File Geodatabase readers as of build 5564.
• An equivalent format attribute for the ArcGIS parameter “Character Width” has been created as of build 5531. It is called geodb_text_character_width
• The personal geodatabase metadata attribute, fme_num_entries, is also exposed in the file geodatabase reader/writer as of build 5529
• Did you catch Don Murray and Mark Stoakes presenting a pre-conference seminar on Data Interoperability? If not, don’t worry. Safe-hosted ArcGIS Data Interoperability Training will take place in Denver, Houston and Washington DC in the coming months. See the Safe training calendar for more information.

This Edition of the FME Evangelist…
…was written to the tune of “Greetings to the New Brunette” by Billy Bragg.

One of Britain’s best singer-songwriters. This is a slightly bizarre video, but you can’t argue with lines like:

“And if you hadn’t noticed yet, I’m more impressionable when my cement is wet.”
“How can you lie there and think of England, when you don’t even know who’s in the team?”

August 6th, 2008

FME Evangelism Weekly Issue #11

Introduction

This “week’s” issue has important news for Linux, DGN and SQL Server users, plus a number of examples created using FME2008.

Also - this email coincides with the release of a new update to FME2008. This one is build 5204. You can get it from our ftp site at: ftp://ftp.safe.com/fme/beta/fme5204.exe

Although there aren’t any large-scale fixes - just a number of minor items that might affect a small number of users - there is a big enhancement with the ability to Write DGN Tags.

See item #2 below for more information on Tag writing, and check out fmepedia for the full list of updates: http://www.fmepedia.com/index.php/FME2008_Post-CD_Fixes

Cheers

Mark

Contents

1. The first 64-Bit FME!
2. DGN Tag writing in FME2008
3. FME and SQL Server Spatial May CTP (CTP-6 Refresh)
4. Checking for Self-Intersecting Lines
5. Weather Maps
6. VolumeCalculator
7. Neighbor Attribute Average
8. Articles on Safe and FME

1) The first 64-Bit FME!

In case you hadn’t noticed (not many people at Safe did!) the first 64-bit version of FME was released for FME2008. It is for Linux only (lucky penguin) and you can find it on the installation CD and on our FTP site.

Note, this is a true 64-bit version, not just one compatible with a 64-bit machine. The ordinary 32-bit version of FME for Linux (called linux-x86) will run on both x86 and x64 versions of Linux.

This 64-bit version will only run on Linux x64.

Also, where the 32-bit Linux FME is built for RedHat EL4 (x86), the 64-bit version is built for RedHat EL5 (x64).

Tux Picture (c) Larry Ewing

2) DGN Tag writing in FME2008

Anyone paying attention to the “What’s Great” page for FME2009 may have noticed that we have added tag writing functionality to that version of FME. For the non-CAD world, tags are a DGN format means of storing small amounts of attribute data.

On this topic there are two bits of excellent news for all you DGN users:

a) Tag writing has been back-ported into the updated version of FME2008 (build 5204). It was just so requested an item we had to do it.

This is what we’re calling ‘phase 1′ of this functionality. You need to create a list attribute for each tagset, each element in that list becoming a tag when you write the output. These would be the same tags you see when reading a dataset with tags.

There is also an option (keyword) to turn this functionality on and off.

There is an example of writing tags using FME2008 on fmepedia at:

http://www.fmepedia.com/index.php/Writing_DGN_Tags

b) For ‘phase 2′ of tag writing, FME2009 will allow you to add user attributes to your destination feature types (a new experience for you DGN users) and will (optionally) write these ‘attributes’ as tags on your DGN data.

More information on that functionality when a beta build is made available.

3) FME and SQL Server Spatial May CTP (CTP-6 Refresh)

If you read Spatial Ed (the blog page for Ed Katibah of Microsoft) you’ll already be aware that an upcoming release of SQL Server will swap the coordinate order for well-known text (WKT) and well-known binary (WKB) formats from latitude-longitude to the customer requested longitude-latitude.

So how will this affect FME users?

Well, if you upgrade SQL Server to the next CTP (for those not in the know CTP is a fancy acronym for beta version) then you should also upgrade your version of FME to FME2008 build 5200 or newer. You can read the list of updates made to that version on fmepedia and find the download on our ftp site.

Those sticking to the older CTP release of SQL Server should experience no problems, even if you still upgrade your FME. The new FME checks the version of SQL Server and acts accordingly.

4) Checking for Self-Intersecting Lines

As many of you will know, FME has a SelfIntersector transformer to remove self-intersections from lines.

Many users (and a few Safers) had always bemoaned the fact it intersects the data and won’t just check for self-intersections. The only way to do this was with a custom transformer like this one on fmepedia.

However, I’ve just found that the GeometryValidator transformer will test line features and flag as a failure any that self-intersect.

Who knew? Obviously not me.

Right: Yes, this line feature fails the “Simple” test and may therefore self-intersect!

5) Weather Maps

Dmitri Bagh - the “Q” of Safe Software - has been busy playing with web services again.

This time he’s dipped into a JSON service that returns information from weather stations within a user-defined bounding box.

It’s a useful example for anyone tasked with reading a JSON or GeoJSON dataset as it deals with the data using JSONExploder and JSONExtractor transformers. The bounding box parameters can - of course - be created using an existing dataset and the BoundsExtractor transformer, which makes this workspace flexible in terms of the area of interest.

You can find Dmitri’s workspace on fmepedia.

6) VolumeCalculator

Safe pro-services dude Dave Campanas recently came up with a useful custom transformer for calculating volume.

The volume is calculated using a set of 2.5D input features and a user-defined “base elevation”; so for calculating the difference between two surfaces you’d want to calculate their volumes separately (to the same base elevation) then simply subtract one from the other.

The transformer also accepts a boundary feature by which to define the area of interest.

http://www.fmepedia.com/index.php/VolumeCalculator

7) Neighbor Attribute Average

As for me, here’s my contribution to the fmepedia workspace pool: a Neighbor Attribute Average Calculator.

This workspace takes a set of adjoining polygons, examines a user-specified attribute and for each polygon calculates the average value of its neighbors.

One use - and I think this is what the requestor wanted it for - would be to identify spikes and wells in a DEM dataset. If the Z value of a polygon is greatly more or less than its neighbors average then it must be an erroneous value.

Do you have any other uses? Let us know.

http://www.fmepedia.com/index.php/Neighbor_Attribute_Average

8) Articles on Safe and FME

Check out the latest edition of GeoInformatics (p24) for an article by Safe president Don Murray on ‘The Growing Need for Spatial ETL’.

http://fluidbook.microdesign.nl/geoinformatics/03-2008/

Does the cover image look familiar?

If that weren’t enough, Dale also got his writer’s hat on and has two blog postings.

The first is about KML becoming an official OGC specification, and how its impact is comparable to the shockwaves felt when Wayne Gretzky went to play hockey for Los Angeles.

http://spatial-etl.blogspot.com/2008/05/kml-is-ogc-open-specification.html

The second is on the recent Where 2.0 show in San Francisco.

It tells how even if you know nano-formats, you might still not know Jack.

http://spatial-etl.blogspot.com/2008/05/safe-on-location-at-where-20.html

Star-Apic (an FME reseller) are holding a “Managing and Publishing Spatial Data” event at the Ordnance Survey offices in Southampton, UK.

Included in the events are a workshop on “Making the Most of FME”. See the Star-Apic web site for more details.

UK reseller Dotted Eyes will be running a FME event at their offices in Bromsgrove, Worcestershire on 18th June 2008. Email marketing@dottedeyes.com for more information.

Now that Celine Dion has finished her Las Vegas show, fellow-Canadian Dale Lutz is taking over with a three day stint at the Intergraph 2008 Conference. June 2nd-5th are the dates. Booth #805 and a 45 minute talk on Spatial ETL for GeoMedia.

For more upcoming events with Safe Software participation, see our web site: http://www.safe.com/aboutus/events/tradeshows.php

The FME Evangelist - this weekly in a more friendly (and email deliverable) form

FME User Central - a new access point for all FME technical resources…

…lots of things, but mostly the Guardian Football Weekly podcast.

In the latest issue the pod discuss John Terry, Avram Grant’s replacement and the play-off finals.

If football (soccer) isn’t your style then try the BBC’s Friday Night Comedy podcast instead.

May 26th, 2008

FME Evangelism Weekly Issue #9

Contents

1. FME User Conference Follow Up
2. The Generic Writer and Published Parameters
3. Workspace Runner Transformer Example
4. Raster to Vector Centreline Extraction
5. Post-CD Fixes

1) FME User Conference Follow Up

A big thank you to all those users who came to the recent conference and made it such a success. We have a number of downloads available as a follow up:

• You can now download all the presentations from: http://www.safe.com/fmeuc/presentations/index.php
• Much to Tyson’s embarrassment, the FME Idol audition video is now up on YouTube: http://youtube.com/watch?v=EJhK9pWv3ao
• The FME Idol challenges are on fmepedia: http://www.fmepedia.com/index.php/FME_Idol_2008

2) The Generic Writer and Published Parameters

One of the questions from the User Conference advanced training was “in the Generic writer, how do I restrict output to a specific list of formats?” At the time I wasn’t sure you could do this, but then Dale and Don revealed the secret and I got it in a flash of light: Shared Published Parameters.

The key is to create a new independent published parameter (aka custom parameter or standalone parameter) - of type choice - in which you define your list of required formats. The list should use the short version of their name, which is what the generic writer recognizes.
Then in the generic writer, locate the Output Format parameter, right-click it and choose Unpublish Parameter, then right-click again and choose Set to Published Parameter. Select the new independent parameter you created.

Now when you prompt and run the translation, the list of generic formats to write to will consist only of those you included in your independent parameter.

A demo movie demonstrating this technique can be found on the FMEGuru YouTube channel:
http://www.youtube.com/watch?v=T1bxHFvRfHo (according to YouTube stats it had 4500+ viewings in the first 24 hours!)

What’s going to be really good in FME2009 is that there will (hopefully) be a new parameter type that is both a choice and a lookup table - so instead of providing a list of formats in their short name (KML, IGDS, GEODB) a user can create a list of full names (Google Earth KML, MicroStation Design, Personal GeoDatabase) and a lookup table that maps them to the correct short name form. This makes for a more user-friendly parameter selection dialog, without having to do any additional processing.

3) WorkspaceRunner Transformer Example
One of the introductory slides at the recent user conference included the cryptic comments:

• Author batch processing workflows with Workbench
  • No need to write batch scripts in an external language
  • Use metadata available in Workbench environment to control the script

The method - which a number of users have asked about - involves the use of two new-for-2008 functions: The Path format reader and the WorkspaceRunner transformer.

For batch processing, the Path format reads a list of the datasets to be processed, and the WorkspaceRunner calls the workspace in which they are to be processed.

You can find an example of this technique on fmepedia.com The same technique can be used on an FME Server installation using the ServerJobSubmitter transformer.

4) Raster to Vector Centreline Extraction
Although FME doesn’t have a specific tool for converting raster to vector, there are a number of functions that can be used, as Dmitri demonstrates in this excellent example.

Dmitri says…
“Although generally I am quite skeptical about this, there are perhaps some areas where we can be quite good with current FME capabilities. A user suggested they could use it for digital maps that never existed on paper - rasters (maybe in PDF) generated by some software. I tried a map from Google Maps for extracting road centerlines and - if cartographic requirements are not very high - these results are quite acceptable.”

The technique relies on pixels of a known colour, which can be a problem with photo images, but not so difficult on a more standard map image. Problems are caused by anti-aliasing and horizontal labels which introduce noise, but the cleanup is generally less onerous than digitizing everything from scratch.

See Dmitri’s example on fmepedia at:
http://www.fmepedia.com/index.php/Raster_to_Vector_-_Centerline_Extraction

5) Post-CD Fixes
The build number for the CD release of FME is 5197. However, because - inevitably - the 2009 development process will throw up fixes we would like to have been in FME2008, what we hope to do is make available approximately monthly updates to the 2008 release build. I’m not sure what these will be called, but in essence they will be like service packs.

We’ll keep you informed about when these new builds are released, and what issues we are wanting to fix. So far the only problem worthy of note is:
Reading Big-Endian WKB

This problem is a failure in reading Big-Endian WKB data. I understand it shouldn’t be a widespread problem, since any WKB that FME has generated will be Little-Endian. Nevertheless, it seems likely a fix for this will be back-ported to 2008 (probably build 5198).

Brief Notes

• The generic writer was getting some output file extensions wrong. This is fixed in build 5191+
• Problems reading Oracle raster pyramids are now fixed in build 5186+
• Not one of our movies, but see how one user gives a lesson on AutoCAD Map 3D Data Connection Using FME: http://www.youtube.com/watch?v=dZ8qrBq-7qc

This week’s Weekly was written to the tune of…
Belgium’s finest scrabble-playing punk musician, Plastic Bertrand and his 1978 hit, Ca Plane Pour Moi.
This one can really implant itself in your head. If it’s all a bit too fast, then here’s the lyrics in both French and English.

April 7th, 2008