Introduction
Hi folks. Starting now I’m going to try and reduce the number of items per post, but publish more often. I’ll still try and mix up topics so there is something for everyone in each post.
Let me know how well this works out for you all.
This issue has the best FME Server demo I’ve seen(!), some exciting news for users of the scripting tools Python and Tcl within FME, plus an important notice for users of FME on Linux who have a fixed (node-locked) license.
Cheers,
Contents
1) FME Server Demo: Handling Real-Time Location Data
2) Python 2.5: Now included with FME2009 (and more!)
3) FME2008: New screen saver
4) TCL: Updated to v8.5.2
5) Linux FME Fixed Licensing: Important Notice
6) Quick Info
FME Server Demo: Handling Real-Time Location Data
The great thing about FME Server (from an evangelist point of view) is that users can run demos directly from our in-house Server installation, and need to download or install very little else.
With that thought in mind, I’d suggest browsing on over to this fmepedia article, which describes and links to a demo that everyone at Safe is all fired up about: handling real-time location data.
The scenario is one of vehicle (taxi) locations being provided by a continuously updated GPS reading. In actual fact we’re fabricating the data with a Python script to ensure an uninterrupted supply, but a real data source would be little different. A workspace is created to read the data, process it and write it to KML. That workspace is uploaded to an FME Server and registered as a KML Network Link service.
And that’s it! Click the workspace URL and it opens up the KML data in Google Earth. Set the network link refresh rate to 2 seconds and you effectively get a real-time visualization of the data as it is updated.
OK. Maybe you’ve seen similar before, but two things to remember:
- This is FME! Simple and quick. There are only 7 transformers in the workspace, and 5 of them are to do with styling the KML. Plus the whole workspace only took 2 hours to develop. In fact it took longer to write the data-creating script.
- This is Spatial ETL! What we have is just a starting point. Now you can start to add more source data and carry out transformations. For example, add some road network data you have in your SDE database; merge in a GeoRSS feed of live traffic information; throw in some transformers to resymbolize the output, or to do a spatial overlay.
Hope you find this example interesting, and maybe a little inspiring. If you know of a real-life feed of GPS data then why not let us know so we can try it out?
Python 2.5: Now included with FME2009
Effective build 5573, FME2009 now includes a Python interpreter and a subset of the standard Python library. This means that users will no longer have to install Python separately in order to use the PythonCaller transformer or Startup/Shutdown Python scripts.
Notes:
- At the moment, only Windows is supported. Similar support for Unix is planned prior to the FME2009 release (in fact our developer expects implementation by the end of July)
- FME will only use its Python when no other installation of Python can be detected. That means users who do not want the FME Python version (2.5) are free to install another version.
- By “subset” of Python we mean we ship everything except tkinter and bsddb - but we’re confident this will not impact most users.
Additionally!
As of build 5573 there are also changes in how macros are handled using Python.
- The FME_MacroValues dictionary is now initialized by default, so no startup script is required.
- In addition to the FME_MacroValues object in the ‘main’ namespace, we also create a reference in the pyfme namespace, so you can use it from within Python factory modules.
- Macro values can now be accessed as objects via pyfme.macros, for example pyfme.macros.DestDataset
TCL: Updated to v8.5.2
The Tcl interpreter in FME2009 has been updated to version 8.5.2. There is a list of highlights on the Tcl official web site (http://www.tcl.tk/software/tcltk/8.5.tml) but in particular we were impressed by:
- Speed: 8.5 now runs 10% faster than 8.4 with reduced memory usage.
- Bignums: Tcl now supports arbitrary-precision integers, which improves math operations on large integers.
clock command: More robust implementation of command for specifying time, with significant l10n and i18n improvements.dict command: New data structure that allows access to data by value rather than a variable name, which is substantially faster.- Additional improvements: Faster list search, new and improved mathematics procedures, anonymous procedures, new ways to package Tcl extensions, Tcl-level custom channel types, file and line location information for each command, and more…
FME2008: New Screen Saver
Resident graphic designer Jara has just completed a screensaver with an FME2008 theme.
Download it from the fun stuff downloads page on fmepedia to find out what our wacky lizard is up to this time!
Linux FME Fixed Licensing: Important Notice
For reasons of reliability and security, the method of generating Registration Keys for Linux versions of FME has been changed for FME2009. This means that as of build 5574 of FME2009, any machine running a Linux version of FME with a fixed license will require a new license.
For more information see the bulletin on fmepedia.
Quick Info
- Reader/Writer and Function/Factory help files have been updated in build 5573
- We’re reviewing all keyboard shortcuts in Workbench - let me know if there are any you would specifically like to see.
- Beta builds of the FME 32-bit and 64-bit Linux versions are now automatically uploaded to our FTP site when available.
- The FSP (Forest Stewardship Plan) reader and writer has been updated to version 1.2 as of build 5562
- Safe announces partnership with WeoGeo to bring FME Server into the world of cloud computing
This Edition of the FME Evangelist…
…was written to the tune of “The Man In Me” by Bob Dylan.
http://www.youtube.com/watch?v=Y2s8_hCCHg4
This song is played in the intro of one of my favourite movies. Kudos if you know which one.
July 17th, 2008
Introduction
Hi folks. This is my first true posting to the FME Evangelist blog. In fact I’m on vacation at the moment, but can set a date for this to be published automatically. Ain’t technology great?!
Anyway, this issue has a nice mix of FME2009 updates, example workspaces, important notices, and an FME Server demo.
Cheers,
Contents
1) FME2009 Beta: Update
2) Logger Transformer: Feature Type in FFS Output
3) PointDisperser Customer Transformer
4) FME and ArcGIS v9.2 SP5
5) FME Server: Data Streaming Example
6) StringAttributeRounder Custom Transformer
7) VirtualEarthTiler: Spherical Mercator Coordinate System
8) Chopper Transformer: Overhaul for FME2009
FME2009 Beta: Update
Issue #12 noted that we were aware of a number of performance issues with the first FME2009 beta. I’m happy to say that all known issues are now reported as resolved and the current build (5557) back to normal.
Logger Transformer: Feature Type in FFS Output
The Logger transformer already had an example of its use on fmepedia, but with a number of updates since then (July 2006) it’s worth re-visiting this unassuming little transformer.
The main change you’ll find since that example was created is that features written to the Logger are now automatically written to a spatial Log file in FFS format. This is a great for visually inspecting features, rather than trying to interpret their representation in a plain text log file.
The other change is something new for FME2009; the ability to specify the feature type to which to write the spatial log features.
Previously the features sent to a Logger would go into a feature type named after the preceding transformer, for example Tester->Failed.
However, in FME2009 you are able to specify the feature type name, and so can be more descriptive – for example “Bad Address Data”, or “Unused Line Features”.
.
Above: The Feature Type to Log setting in the Logger transformer in FME2009
Visit the Logger example on fmepedia to see the updated version of the workspace.
Did You Know?
Did you know that FFS datasets come in both file and folder based flavours?
For me, the Logger transformer is particularly nice to use because multiple Loggers in the same workspace write their data to the same file-based dataset.
Compare this to the FFS format writer, where different feature types write to different FFS files within a folder (folder-based dataset).
There are benefits to both, but my preference is for a tidy, single-file dataset, so I choose the Logger every time to write my FFS files.
Below: The Logger transformer produces a file <workspaceName>_log.ffs…
…and that single FFS file contains two different feature types (Invalid Day and Invalid Month):
PointDisperser Custom Transformer
The PointDisperser custom transformer is a tool to create a form of structure suitable for thematic mapping. Where a number of points all belong to a certain parent location, the transformer can disperse them around the parent in a sort of radial diagram.
With a little more work the points and lines can be easily turned into coloured nodes and sectors.
These features look very effective when overlaid on a map background using some form of mapping or visualization tool.
In line with the concept of “eating one’s own dog food“, at Safe we use this tool for mapping sales figures, where the parent point represents a country or city, and each node represents an FME sale at that location.
No doubt you could envisage similar uses for your organization.
FME and ArcGIS v9.2 SP5
This is an important note for users of the Geodatabase readers and writers who upgrade their ArcGIS v9.2 installation to SP5.
FME 2007 and 2008 – which were both created before the release of SP5 – are both adversely affected by this upgrade. In particular you may receive a specific error message:
“The procedure entry point SgCodeFree could not be located in the dynamic link library sg.dll.”
Of course this will also impact any SpatialDirect and FME Server installations that use these readers and writers.
The quick-fix we have for this problem is to copy a number of files from the ArcGIS installation, and paste them into the FME installation folder, overwriting files of the same name.
The files are:
No problems have been reported using this workaround, but we don’t guarantee it because it’s a newer library than what FME was compiled against.
The long term fix is for us to issue updated versions of both FME2008 and FME2009, which we intend to do in the near future. If you want to be notified when this happens please email support@safe.com and quote PR#15697
NB: Only update the DLL files if you get the above error message. They should not need to be changed for any other reason.
FME Server: Data Streaming Example
Data streaming is a service provided by FME Server that delivers data right into an external application.
This example - which you may have seen as a demonstration before - is a very good illustration of that technique, and one of our favourites here at Safe.
Source data - in this case a live feed of Earthquake data from the USGS (GeoRSS format) - is translated into KML using a workspace that also restructures the data from simple points into extruded columns.
By uploading the workspace to FME Server the output data can be streamed directly into Google Earth – by anyone! The end user does not need FME to be able to visualize this data as all of the processing is done on an FME Server installation.
You can do this too! By simply clicking on this URL (or entering it into a web browser):
http://fmeserver.com:80/fmedatastreaming/translate/Samples/earthquakesextrusion.fmw
…you’ll make a call to an FME Server, which will carry out a translation, convert the data to KML, and stream it straight into Google Earth on your computer.
Above: The output viewed in Google Earth
The source data doesn’t need to be a live feed (as it is in this example). It can be any source data that you have on your system and that FME Server has access to - for example your spatial database (SDE, Oracle, SQL Server, etc). This means any person in your organisation can view your data - in Google Earth - immediately, and without you having to translate it for them.
Also, you can use more than one source dataset, in effect creating a Google Earth mashup with very little effort.
For example, you could merge a set of delivery routes with a live traffic alert feed (http://developer.yahoo.com/traffic/rss/V1/index.html) to show where a delivery schedule could be interrupted by traffic congestion.
StringAttributeRounder Custom Transformer
The AttributeRounder transformer is great for rounding the values of attributes (for example 123.456789 to 123.46) – but not so good when the attribute contains some sort of alphabetic character (for example 123.456789m). But rounding this form of text can be a requirement when the data is annotation read directly from a cartographic input.
The StringAttributeRounder is a custom transformer created in order to allow users to round off numeric attributes that are contained within a longer string.
The user can choose whether the text part of the string is at the start or end of the attribute.
For example:
Start Attribute: m234.5555
Text Part At: Start
Decimal Places: 2
Result Attribute: m234.56
Start Attribute: 234.5555m
Text Part At: End
Decimal Places: 2
Result Attribute: 234.56m
The transformer also handles <space> characters, for example:
Start Attribute: x y234.5555
Text Part At: Start
Decimal Places: 2
Result Attribute: x y234.56
If you want to use the transformer, or are interested to see how it works, then visit the StringAttributeRounder page on fmepedia. It requires FME2008, though there is also a version specifically for FME2007.
Above: In brief, the transformer strips off the text part of the string (from either the start or end), rounds off the remaining number, then glues it all back together with a Concatenator.
There is one specific best practice technique to look out for; the AttributeExpressionRemover transformer is used to clean up any temporary attributes created during the process, so the user is not distracted with extra data on the output.
VirtualEarthTiler: Spherical Mercator Coordinate System
A couple of users have experienced a problem with the VirtualEarthTiler in the last week, so I wanted to try to explain the cause (neither obvious nor simple), how to workaround it in FME2008, and how it has been resolved with FME2009.
The VirtualEarthTiler is used to turn a raster feature into a number of smaller tiles for use in Virtual Earth. It essentially does the same job as the MSR MapCruncher, but without the registration process.
The problem experienced was that the tiles that got written were not square; instead of being 256×256 they would be something like 400×256. To get square tiles required the ‘World File’ option to be turned on, posing the puzzling question “why should a world file make any difference to the size of the actual raster file itself?”
There are in fact two issues causing this to happen:
Issue One
Tiling data for VE (or GE) is more complicated than you’d think. Because source data is in plain LL and Virtual Earth uses a spherical coordinate system the output ends up with a latitude distortion. This distortion exhibits itself as non-square pixels in the raster output.
For example, in a test workspace I sent the VirtualEarthTiler output to a Logger and got this:
Number of Rows : 256
Number of Columns : 256
Cell Spacing : 0.0003433,0.00029677
As you can see, the cell spacing is different making these cells/pixels non-square in shape.
However the PNG format (as with many other raster formats) only has a single “field” in which to record cell spacing; i.e. it is assumed that X spacing is the same as Y. Therefore, because the PNG writer can’t write non-square cells, the data is resampled to make the cells square.
It’s this resampling that causes the overall size of the file to be changed from the expected 256×256.
Issue Two
FME resamples the data to create square cells because, if it did write non-square cells, how would another application know that there is odd spacing?
The answer is the world file.
That’s why writing a world (WLD) file makes a difference to the output.
FME is happy to write non-square cells provided there is a world file, because the world file tells other applications what the cell spacing is.
So, if you do write the world file then the cells will be non-rectangular, but you MUST keep the WLD file with the PNG else any application that reads the PNG will not know what the cell spacing is and so will not show the raster in its correct size and location.
FME2008
So, when you are using FME2008, you MUST have the ‘write world files’ setting turned on in your writer, and you MUST store the world files alongside the tiles for Virtual Earth to interpret the data correctly.
FME2009
Getting to the real point of this topic, FME2009 supports a new coordinate system called the Spherical Mercator, aka Web Mercator, EPSG:3785 or EPSG:900913*
Spherical Mercator is the coordinate system required for correct tiling of data. So convert your source raster to this coordinate system using the Reprojector, pipe it into the VirtualEarthTiler and you will get… tiles with perfectly square cells; without the need for world files either.
Problem solved.
* Get it? No neither did I for the longest time. I must be a n00b.
Chopper Transformer: Overhaul for FME2009
Many people who use the Chopper transformer do so to convert lines into points. However, did you know it can also be used to chop polygons into smaller area features? This might be useful where you have a maximum number of vertices requirement for your destination data.
Simply feed polygon features into the Chopper, and use a chop value of x (4 or greater), and the polygon will be split into a number of smaller area features. Each feature will have a maximum of x vertices.
Above: In this example the Chopper vertices parameter was set to 6, so no output polygon has more than 6 vertices.
This isn’t new functionality in itself, but what is new is that as of build 5556 in FME2009 beta the Chopper transformer has undergone a major improvement in the algorithm used to chop polygons.
The improvement results in:
- Faster Computation
- A Reduced number of output features
- More visually appealing output
For example:
Above: A polygon chopped using FME2008
Above: The same dataset chopped with FME2009
This Edition of the FME Evangelist…
…was written to the tune of “It doesn’t have to be this way” by the Blow Monkeys.
http://www.youtube.com/watch?v=co8P6fN732U
I don’t know how I came across this one, but it brought back some memories. 80’s music videos eh? Nothing like ‘em.
June 25th, 2008
RSS 2.0