The FME Evangelist

FME Evangelism Weekly Issue #6

Contents

• Format Names
• FME Worldwide User Conference – Sneak Peek
• New Format: Golden Software Surfer 6 Binary Grid
• SpikeRemover Example
• HTTP Authentication Function
• FME for FDO
• Jason Birch’s list of favourite FME Transformers!

1) Format Names

For FME2008 you may notice some changes in the names of formats, as we have implemented a set of guidelines to try and standardize format names, and prevent them having to be continually updated.

In general we have dropped version numbers from format names, because it only gets confusing as to what is supported, plus the names need to get changed when a new version comes out. Also we’ve avoided using the words “Database”, “File”, or “Raster” unless it is really part of the name.

Acronyms will not get expanded, unless they are rare enough to need explanation, and some format names will be in a non-English language where appropriate.

2) FME User Conference - Sneak Peek
Having spent all of yesterday eavesdropping on the marketing team, I can exclusively(!) reveal the latest inside information about the upcoming user conference.

The latest session to be added to the agenda is:

Do or Do Not: Master Yoda’s Guide to Avoiding the Dark Side of Raster
As the abstract says, “Powerful is FME; but complex are raster formats. Great are the differences between a raster band and palette. Join us to find out how to avoid the pitfalls faced by an apprentice raster master, and how to effectively translate data between raster formats.”

Also revealed are the titles of the hands-on workshops:

• FME Server
• Updating Spatial Databases with FME
• Tips and Tricks to Make You an FME Superhero

Click these links for the current conference agenda and full list of presentations …and don’t forget that early bird registration ends in a week’s time.

3) New Format: Golden Software Surfer 6 Binary Grid
One truly new item this week is the addition of a reader and writer for the Surfer Binary Grid format, version 6. Apparently this is a format used in the petroleum and exploration industry.

Technical Info: The format stores a single band of floating point numerical (Real32) data, and has the ability to set a nodata value. Surfer 6 Binary Grid files also store some georeferencing information (minimum and maximum x, y, and z values). However, they do not support attribution or coordinate systems, except via external metadata files such as .TAB files.

The format is available in the latest FME2008 betas, and requires a Professional Edition license or higher. Support for version 7 and the version 7 ASCII variant will follow in FME2009.

4) SpikeRemover Example
The SpikeRemover transformer is a new item that….. well it removes spikes from your geometry.
Now we have a good example of that transformer on fmepedia, courtesy ofFME scenario specialist Dmitri Bagh.
http://www.fmepedia.com/index.php/SpikeRemover

Dmitri notes that: “If the geometry of a feature is a path, the transformer removes spikes between consecutive path segments as well. For a polygon or donut, if the start/end point is a spike, then it is also removed. The end result is still a polygon/donut. Any polygons, donuts, paths or lines that are part of a collection of geometry [ie aggregate] will also be processed.”

5) HTTP Authentication Function
All web-related formats and transformers such as WFS (left), WMS and the HTTPFetcher have recently been enhanced with the option to use HTTP Authentication. This means that users can now access data that is protected by a username and password.

Three types of authentication are supported: basic, digest and NTLM

6) FME for FDO
In the category of “nice FME use we wouldn’t have known about if the user didn’t write a blog” here are a couple of articles about the new FME for FDO application.

Firstly, user Sean Twomey used this tool to read a whole folder of MapInfo TAB files in a way that’s both “live and funky”. Thanks for the great review Sean. http://edgeofwedge.com/survey.php?p=46

Secondly, user Gwenael Bachelot also reads a folder of MapInfo TAB files (I spot a trend here), and helpfully points out that a full FME license would expose within the FDO all of the other FME supported formats. His article is in French, but with a handy button to translate to English. Thanks Gwenael.
http://geospatialfrance.typepad.com/geospatialfrance/2007/12/fournisseur-fdo.html

7) Jason Birch’s list of favourite FME Transformers!
In case you don’t read the user group postings, Jason Birch (Certified FME Professional, MVP and all round FME power user) recently listed his favourite transformers. I don’t remember why, but it’s an interesting read:

Best Transformer
TopologyBuilder - allows some amazing Q/A processes when combined with FeatureMergers SpatialRelator - relationships are the backbone of complex spatial analysis ExpressionEvaluator - swiss army knife of transformers
Best Supporting Transformer
StringSearcher - insanely powerful for pattern matching and fuzzy searching Tester - this simple but solid transformer, along with its close cousin AttributeFilter, is used in almost all of my workspaces Concatenator - used extensively to build “maptip” fields for web maps FeatureMerger - flip and fold features like there’s no tomorrow
Best Transformer Producer
Sampler - speeds up development, slashing production costs Visualiser - provides insight into production on a granular level SummaryReporter - while this is good for ongoing reporting, it can also help considerably during production
Best Transformer Category
List* - lists are my absolute favourite construct in FME Raster* - very, very handy when you need them
Coolest Transformer Not Used
TransporterSender/Receiver - these things are neat, but I don’t have the volume that would benefit from their use
Least Understood Transformer
SchemaMapper - I know this could be powerful, but have never taken the time to figure it out

Jason also says:
“Of course, a number of transformers and other components require special merit awards. This includes things like neo-readers and writers like KML, GeoRSS and GeoJSON, and neat things like PythonCaller (and its cousin TCLCaller), WebCharter, KMLStyler (wow, what a lifesaver this thing is), VirtualEarthTiler, and HTTP* transformers.
OK, forget the list. I just love all of FME :) “

Brief Notes

• The Extruder transformer has been upgraded to extrude point features into a vertical line
• The IFC Reader now has added styledItems support
• The AutoCAD OD reader has a fix to handle the case when arcs are split on reading
• The time taken to open large workspaces should be cut by a performance improvement in FME2008.
• The RasterGeoreferencer transformer now prohibits the use of negative settings values

This week’s Weekly was written to the tune of…
Ian McNabb’s “Still Got the Fever”. The whole album this is from (Head Like a Rock) is full of great tracks like this.

This one isn’t on iTunes (other McNabb songs are), so here’s a YouTube link of a live version. http://www.youtube.com/watch?v=KjADocr31Go

January 18th, 2008

FME Evangelism Weekly Issue #5

Contents

1. Reprojector “Read from Feature” Setting
2. Creating Pivot Tables with FME
3. Batch Translation and Source Name
4. Second Order Conformal Transformation
5. AreaBuilder and Lists

1) Reprojector “Read from Feature” Setting
The Reprojector transformer recently received an upgrade in the form of a setting to read the source coordinate system from a feature, rather than setting it as a constant. This means that you can run multiple source datasets - all with different coordinate systems - through the same Reprojector.

Gotchas:
a) If you change the setting from “Read from Feature” to an actual coordinate system, you can’t revert back again. You would need to replace the transformer.
b) The source data must be tagged with a coordinate system. This will happen automatically for a format that supports coordinate system definitions.

To see a live example of this, a movie has been created and uploaded to YouTube.
http://www.youtube.com/FMEGuru

2) Creating Pivot Tables with FME
Pro Services dude Aaron recently came up with a way to use the new group-by option in the StatisticsCalculator to create Pivot tables.

A pivot table is a summary of a group of records (hence the use of the group-by option) within a database or spreadsheet. The ability to create these has been a much requested function in FME.

The top image is a pivot table generated by Excel, the lower image a pivot table generated by FME.

Aaron’s workspace is available for download from fmepedia at:
http://www.fmepedia.com/index.php/Pivot_table

The really nice part is how the workspace is extended to use the new WebCharter and ImageFetcher transformers, to visualize the output thus:

3) Batch Translation and Source Name
The Batch Deploy wizard in Workbench is a tool used to batch process data.
The output dataset name is always taken from the source name, for example: Roads.dgn would become Roads.gml

However, this doesn’t always make sense when using folder based datasets, for example:
C:\myData\ShapeFiles\roads.shp
converted to MIF/MID and pointed to a MIFFiles folder would become…
C:\myData\MIFFiles\ShapeFiles\roads.mif

That’s because the “ShapeFiles” part is treated by FME as the name of the source dataset, and so is included in the output.

In recent FME2008 betas the batch wizard has an option to drop the source dataset name from the output:

By unchecking this box (which for backwards compatibility is checked by default) the source dataset name is dropped, giving (with the above example):
C:\myData\MIFFiles\roads.mif

Of course the important thing is to be aware of the difference between file and folder based datasets, and how they all work, else you won’t know whether to check the box or not anyway!

4) Second Order Conformal Transformation
FME2008 now has a new transformer called the SecondOrderConformer.
The best way to explain this topic is to perhaps give a little background (as I understand it… and I could be wrong)!

There are a number of basic geometric transformations you can carry out on spatial data: rotate, scale, reflect etc. FME does all of these.

Below is an example of a simple rotation. The green grid of squares is the original.

A slightly more advanced method is called an Affine Transformation. FME does this too. In effect it is a linear transformation (as is the simple example above) in which the x and y dimensions are independently transformed. A simple example is shown below, where the pink grid is the original data. Notice how straight lines remain straight, and parallel lines remain parallel.

An even more advanced method is the second order conformal transformation, aka a polynomial transformation, which is what we have just implemented. This is a non-linear, variable scale transformation (meaning that lines will not necessarily remain straight or parallel), an example of which is shown next:

Are you with me so far? Good, because now you know as much about the theory as I do.

Gotchas:

a) A true transformation would mean turning a line into a curve. But if FME does this, any topologically connected linework would be broken apart. So FME will only carry out transformations on the vertices of a feature. The image below shows what happens when you transform a simple polygon feature (it doesn’t curve properly).

To get more exact curves you should use a Densifier beforehand to add extra points.

b) True 3D geometries (faces etc) are not supported. They get output through an ‘invalid’ features port.
Of course, the big question is why would you want to do any of this? The answer is that you want to transform your data into a known space, for example to transform data into a local grid system. A reprojection doesn’t work because the local grid is usually one whose coordinate system is not known. The only known things are a set of control points common to both systems, from which the transformation parameters can be calculated.

The technique is also used for adjusting aerial photos and satellite images and, according to a Google search, in the medical world for aligning brain scan images (though I’m not aware they use FME!)

5) AreaBuilder and Lists
The AreaBuilder transformer takes line features and turns them into polygons. A recent update to the AreaBuilder transformer provided it with an option to record attributes in a list. This solves the problem of how to retain all of the different attributes from the line features that made up the polygon.

For example, we could take a map of Lake Victoria (below), turn lake line features into a polygon, AND keep a list of the names of the surrounding countries (assuming that the lines contained this information). The answer is Uganda, Kenya and Tanzania in case you were wondering.

Our resident FME trailblazer, Dmitri, made good use of this ability in a custom transformer called the MultiLayerAreaGeneralizer. This transformer - available on fmepedia - generalizes multiple layers of data without getting confused as to which data belongs to which layer.

See http://www.fmepedia.com/index.php/MultilayerAreaGeneralizer for full details.

Brief Notes

• The 3DExtrusionCreator transformer has been renamed to Extruder, to simplify the name.
• The 3DFaceCreator transformer has been renamed to FaceReplacer, since it is more of a replacer type transformer.
• VRML Writer now has support for 3D geometries
• The 3DExtrusionCreator will now handle 2D features - base elevation is assumed to be zero
• Workbench Drag and Insert behaviour has been updated so that the dialog box does not appear if there is only one option

www.safe.com/fmeuc

January 11th, 2008