Contents:
1) Raster To Vector Coercing
2) FFS Support for Raster Data
3) Visualizing Rasters
4) Persistent Cache and Raster
5) Transporting and CheckPointing Rasters
6) New Raster Formats
7) Raster Cell Origins
8) Raster Performance
9) Even more raster stuff…
Introduction
Hi FME’ers
This edition of the FME Evangelist is a special raster edition.
I know I wanted to vary topics in each post, to give something for everyone, but there’s so much raster info to go over that I can compile it into a single posting and still have enough left over to sprinkle into future posts. I have to say that our raster support is now very solid in FME, and well worth taking a look if you’ve not used it before. In particular I think being able to combine vector and raster data together, especially in FME Server applications, is going to be a big hit.
Regards,
Raster To Vector Coercing
Vectorizing raster data is a useful function that has just undergone a name change: RasterCellCoercer is the new name for the RasterToPointCoercer transformer. The reason it has been renamed is a very good one: the transformer now supports coercing raster data into polygons as well as points.
In the new ‘polygon’ mode a single polygon is output for each cell in the original raster. This results in a large number of polygon features - potentially useful in itself - but which you could dissolve together (based on the original cell value) to create a more manageable and realistic vector dataset.
In this example on fmepedia (screenshots below) a raster DEM dataset is converted into vector polygons using exactly this method.
FFS Support for Raster Data
I’m pleased to announce that as of FME 2009 beta build 5568 the FFS format is able to store raster features.
Rasters stored to an FFS file will have their data written to a corresponding .frs (FME Raster Store) file. One FRS file may hold the data for multiple raster features. FRS files hold up to 2GB of raster data; if the raster data surpasses the 2GB file size limit the data is automatically split across multiple files.
Above: A CDED raster dataset converted to FFS
Raster FFS datasets are used in FME (e.g. opened in the FME Viewer) by selecting the FFS file [though I tried opening the frs file and it did work - I just had to manually select the FFS format]. If you select an FFS file and the corresponding FRS file is missing then FME will just use the bounding box of the raster.
OK - writing raster to FFS might not seem like your idea of a good night out; however, it does integrate raster better with FME and open up some exciting benefits…
Visualizing Rasters
Did you see the previous FME Evangelist? If so, you may remember that:
“When you send data from Workbench to Viewer using a Visualizer transformer, what happens is that the data is simply written to a temporary FFS format dataset which is subsequently opened in the Viewer.”
In older versions of FME you’ve not been able to send raster data directly from Workbench to the Viewer - all you get is the dataset outline - because in between the data is written to FFS, and FFS doesn’t support raster.
But wait a moment! Hold the front page! FFS does now support raster.
So you are now able to direct raster data through the Visualizer transformer and get the correct view. One of those features a new user might say “yeah, so what?” about, but which an existing user will see as a great improvement.
Persistent Cache and Raster
Ever seen the FME Viewer option View > Options > View Persistent Cache and wondered what it does? Well turning it on causes all the data you open in the Viewer to be saved as FFS data in a temporary directory, so that every time you refresh the view or query the data you’re working on a local copy. That way you’re not having to re-read the source data, which can be a lengthy process if it is a web or database dataset.
Obviously this excludes raster, because raster can’t be saved as FFS. But wait! Hold the… er second page! FFS does now support raster.
So now using the Persistent Cache option in Viewer becomes a valuable time saver when inspecting raster data. The FME Viewer can even be shutdown and started at a later date and retain all the local copies of the data and thus the rapid access to the raster data.
The one word of warning is that this will create large amounts of data in the local temporary directory, for example, “C:/Documents and Settings/<username>/Local Settings/Temp/FMEUniversalViewer” - so make sure you have a lot of disk space, and try use your fastest disk.
Transporting and CheckPointing Rasters
Wait! Stop…. actually don’t stop the press on account of these functions; not many users are likely to have heard of these - let alone use them. But the ability to write raster to FFS also:
- Permits support for raster data in the TransporterReceiver and TransporterSender transformers.
- Lets the RasterCheckpointer transformer save the current state in FFS (rather than GeoTIFF) which means that any aspect of raster data expressible in FME can now be accurately stored on disk and restored without data loss.
…and trust me, these are important updates, even if part of slightly less used features.
New Raster Formats
FME 2009 has new support for a couple of raster formats; the first of which may be of interest even to folk who are not generally raster users.
The Shuttle Radar Topography Mission (SRTM) Height format (short name SRTMHGT) is DEM data created by NASA. FME now has a reader for this format. A writer is not planned. The nice thing is that data in this format is freely available at NASA’s repository (though it wasn’t working when I tried it last)
Image Courtesy NASA/JPL-Caltech.
The data comes in two levels: SRTM-1 with data sampled at one arc-second intervals in latitude and longitude, and SRTM-3 sampled at three arc-seconds.
Datasets are divided into individual height files that each contain one by one degree latitude and longitude tiles. Height file names refer to the latitude and longitude of the lower left corner of the tile, for example, N37W105 has its lower left corner at 37 degrees North latitude and 105 degrees West longitude. Height files have the extension ‘.hgt’ and express the height values in meters. These files always have the LL84 coordinate system.
One gotcha is that FME is currently unable to read height files that do not conform to the specified file naming convention, since the file name is the only means of reading the proper referencing for the file.
ER Mapper ERS format (short name ERS) is the internal raster format used by ER Mapper. FME now has a reader AND writer for this format.
The ERS format consists of a header with a .ers extension and a separate raw binary data file that can have any extension. The header file is considered the dataset in FME and contains information about the contents of the binary data file.
Limitations are that the ERS writer currently cannot write nodata values, and that the reader does not support ERS files containing algorithms for image creation and manipulation.
Raster Cell Origins
As you know, a raster is made up of cells, each of which is a single point but which in fact covers a small area. What’s less known (perhaps) is that each cell has an origin point which identifies the representative location of each cell. The origin is an x/y pair between 0.0 and 1.0 inclusively. The default value in most cases is (0.5,0.5) indicating the center of the cell.
Cell origin is particularly important when the cell value is taken from a single position - when the value is the average value of the area the cell covers then cell origin is less important.
FME2009 includes new functionality to set a raster’s cell origin to specified X and Y values. The transformer is the RasterCellOriginSetter and the underlying function is @RasterCellOrigin
Why would you need to do this? Two reasons spring to mind.
Some formats of raster data don’t store cell origin; it is assumed to be a certain location (often upper-left corner, which is counter-intuitive for many map users). So this new tool allows a user to properly adjust data read from those formats, if they so desire.
The second use case is during raster-to-vector, or vector-to-raster, operations. In these scenarios you want to be sure that cells converted to points (or vice versa) end up in the correct location. If you don’t correctly set the cell origin you could find the resulting output is out by up to half the width of a cell.
If you don’t know what the current cell origin is, you can retrieve that information into attributes using the RasterPropertiesExtractor or by inspecting data using the Logger transformer.
Raster Performance
If you’ve not tried an FME2009 beta yet, it might be worth doing just for performance reasons.
Our raster team tells me that several common raster operations take less time to execute in FME2009.
Specifically, raster resampling and rotation should now be faster in all cases. The magnitude of improvement may vary from translation to translation, but these operations should be approximately 30-50% faster than in previous versions.
Raster reprojection benefits primarily when dealing with multi-band rasters. The common case of reprojecting a 3-band raster is now up to 50% faster.
Even more raster stuff…
If you have used FME2009 you may well have noticed the new RasterExpressionEvaluator transformer, and the new Raster clipping capabilities. Since they are so important I’ll talk about them another time: for now check out Dmitri’s RasterExpressionEvaluator examples for some absolutely stunning uses of this transformer, and try for yourself clipping raster features into non-rectangular shapes (very simple and effective).
This Edition of the FME Evangelist…
…was written to the tune of “Only You” by Yazoo.
I’ve been re-watching The Office (UK version) and the great Tim/Dawn romance. Here’s the end of the second series, and here’s the end of the following Christmas special where “Only You” is the background music.
Or, if you only want the music itself, here’s the original.
July 31st, 2008
Contents
1) Workspace Example: CSV Writing
2) Transformer Updates: The Sampler, The Visualizer, His ListSearcher, Her SurfaceModeller
3) Safe Science Lab: Multi-Threading
4) KML Format: Where’s Google?
Introduction
Hi FME’ers.
If there’s a common theme to this issue, it’s how little changes - or little bits of knowledge - can have a disproportionately large impact on the performance of our users. The section on transformer changes in FME2009 is a good example. Each one of the highlighted updates was initiated by a user request, and is a relatively small enhancement that is a big saver in either performance or user effort.
And that’s why we have this technical bulletin: to spread the word about these updates and, by doing so, maximize our users’ performance. That way we’ll give everyone the chance to look their boss in the eye and say modestly “Don’t thank me. We’re all part of the same team”.
Cheers,
mark.ireland@safe.com
PS: Notice the slight change in format, with the expanded title and contents section first - that’s to give a better experience for our RSS subscribers. Hope it works.
Workspace Example: CSV Writing
One of the most common support questions is how to turn plain text x/y coordinates into proper spatial data, but sometimes we get the reverse question: “how do I write out a set of spatial features as a file of x/y coordinates?”
As with many FME tasks, achieving this requires a sequence of transformers; the snag being that the first of these - the CoordinateConcatenator - is relatively unknown. What this transformer does is extract the coordinates of a feature as a long string of (comma separated) numbers and attaches them as an attribute. From there it’s a relatively simple task to split off each coordinate pair into a list (AttributeSplitter), explode that list into a separate feature per pair (ListExploder) and then write the data to a file using the Textfile writer.
A second complication is that quite often you’ll want a header record in front of each features’ coordinates, in order to identify each feature within the text file. This is where you need to get creative in adding an extra “stream” to your workspace - attendees of the Advanced Training at the 2008 FME User Conference will understand what I’m talking about - and making sure that everything gets back into the correct order.
When you’ve done all that the final workspace will look something like this:
But don’t worry if you can’t quite understand what’s happening - it’s my fault for not using Best Practice - because the workspace and detailed explanation are now available as an fmepedia workspace example. Hope you find it useful.
Transformer Updates
Recent FME2009 builds have seen a number of very useful transformer updates.
Sampler
The Sampler transformer now has an extra output port through which non-sampled features are output. This could be useful in a number of situations, but particularly where you want to skim off the first x features while retaining the rest.
Visualizer
When you send data from Workbench to Viewer using a Visualizer transformer, what happens is that the data is simply written to a temporary FFS format dataset which is subsequently opened in the Viewer. A recent update to the Visualizer allows you to specify the name and location of this FFS file, meaning that a user can save a more permanent copy of the output without having to add a writer.
ListSearcher
Previous versions of the ListSearcher allowed the search string to be specified by an attribute value by using the format &<attributename> . A recent update to the ListSearcher turned the setting into a combination of text entry and drop-down list (what we at Safe call STRING_OR_ATTR), meaning that the same effect can now be achieved by selecting an attribute directly rather than having to enter the FME-specific syntax.
SurfaceModeller
Surface modeling operations in FME work by converting a number of input features (3D lines, points, breaklines, etc) into an internal FME Surface Model from which various outputs such as contours and triangles can be generated. Considerable work has to take place to create the initial surface model, which leads to large overheads when the same surface model is required in different workspaces.
A new option in the SurfaceModeller reduces this overhead by allowing the user to read and write FME Surface Model datasets (.fsm files). Now a user can save the surface model from one workspace for use in another; a useful benefit when (for example) needing to drape different sets of features over the same surface.
Above: When this translation re-used a surface model it was over 50% quicker.
I’ve asked to have the GUI adjusted slightly (to me it’s not quite intuitive enough) so in the meantime my advice is to read the SurfaceModeller help very carefully. Plus, I’m hoping to find the same options exposed in related transformers (such as the SurfaceDraper) in the near future.
Above: Overwrite mode creates a new surface model file. Append mode reads the data back - when processed it will write it back with any additional features that entered the transformer appended into the model.
Safe Science Lab
The latest news from Safe’s top scientists is that work is slowly beginning on applying multi-threading to FME operations. For those not in the know, a thread is a computer task, so multi-threading simply means executing multiple tasks at the same time. For example, measuring the length of ten line features could be achieved by ten separate, but simultaneous, tasks, rather than one task which processes each feature in order.
It sounds easy, but the analogy of juggling ten balls at once - instead of one ball ten times - holds good: it’s a much more complicated manoeuver and we certainly want to practice before going live in public! So expect small steps on this one, rather than a single, massive improvement.
KML Format
As of build 5572 the format “KML21 (Google Earth KML)” has been renamed “OGCKML (OpenGIS® KML Encoding Standard)”. Two obvious reasons: 1) OGC now ‘own’ the format and 2) the ‘21′ part was confusing since our KML support is up-to-date and we do actually support version 2.2.
The big gotcha here is for users who type “Google” into the formats gallery - since Google has been removed from the format name you’ll no longer find the format. Type “KML” and you’ll have more success.
This Edition of the FME Evangelist…
…was written to the tune of “Taking on the World” by Amsterdam.
Amsterdam are a great band from Liverpool. Check out this live performance, then go buy their latest album.
http://www.youtube.com/watch?v=oLk7NY0f1_o
July 24th, 2008
Introduction
Hi folks. Starting now I’m going to try and reduce the number of items per post, but publish more often. I’ll still try and mix up topics so there is something for everyone in each post.
Let me know how well this works out for you all.
This issue has the best FME Server demo I’ve seen(!), some exciting news for users of the scripting tools Python and Tcl within FME, plus an important notice for users of FME on Linux who have a fixed (node-locked) license.
Cheers,
Contents
1) FME Server Demo: Handling Real-Time Location Data
2) Python 2.5: Now included with FME2009 (and more!)
3) FME2008: New screen saver
4) TCL: Updated to v8.5.2
5) Linux FME Fixed Licensing: Important Notice
6) Quick Info
FME Server Demo: Handling Real-Time Location Data
The great thing about FME Server (from an evangelist point of view) is that users can run demos directly from our in-house Server installation, and need to download or install very little else.
With that thought in mind, I’d suggest browsing on over to this fmepedia article, which describes and links to a demo that everyone at Safe is all fired up about: handling real-time location data.
The scenario is one of vehicle (taxi) locations being provided by a continuously updated GPS reading. In actual fact we’re fabricating the data with a Python script to ensure an uninterrupted supply, but a real data source would be little different. A workspace is created to read the data, process it and write it to KML. That workspace is uploaded to an FME Server and registered as a KML Network Link service.
And that’s it! Click the workspace URL and it opens up the KML data in Google Earth. Set the network link refresh rate to 2 seconds and you effectively get a real-time visualization of the data as it is updated.
OK. Maybe you’ve seen similar before, but two things to remember:
- This is FME! Simple and quick. There are only 7 transformers in the workspace, and 5 of them are to do with styling the KML. Plus the whole workspace only took 2 hours to develop. In fact it took longer to write the data-creating script.
- This is Spatial ETL! What we have is just a starting point. Now you can start to add more source data and carry out transformations. For example, add some road network data you have in your SDE database; merge in a GeoRSS feed of live traffic information; throw in some transformers to resymbolize the output, or to do a spatial overlay.
Hope you find this example interesting, and maybe a little inspiring. If you know of a real-life feed of GPS data then why not let us know so we can try it out?
Python 2.5: Now included with FME2009
Effective build 5573, FME2009 now includes a Python interpreter and a subset of the standard Python library. This means that users will no longer have to install Python separately in order to use the PythonCaller transformer or Startup/Shutdown Python scripts.
Notes:
- At the moment, only Windows is supported. Similar support for Unix is planned prior to the FME2009 release (in fact our developer expects implementation by the end of July)
- FME will only use its Python when no other installation of Python can be detected. That means users who do not want the FME Python version (2.5) are free to install another version.
- By “subset” of Python we mean we ship everything except tkinter and bsddb - but we’re confident this will not impact most users.
Additionally!
As of build 5573 there are also changes in how macros are handled using Python.
- The FME_MacroValues dictionary is now initialized by default, so no startup script is required.
- In addition to the FME_MacroValues object in the ‘main’ namespace, we also create a reference in the pyfme namespace, so you can use it from within Python factory modules.
- Macro values can now be accessed as objects via pyfme.macros, for example pyfme.macros.DestDataset
TCL: Updated to v8.5.2
The Tcl interpreter in FME2009 has been updated to version 8.5.2. There is a list of highlights on the Tcl official web site (http://www.tcl.tk/software/tcltk/8.5.tml) but in particular we were impressed by:
- Speed: 8.5 now runs 10% faster than 8.4 with reduced memory usage.
- Bignums: Tcl now supports arbitrary-precision integers, which improves math operations on large integers.
clock command: More robust implementation of command for specifying time, with significant l10n and i18n improvements.dict command: New data structure that allows access to data by value rather than a variable name, which is substantially faster.- Additional improvements: Faster list search, new and improved mathematics procedures, anonymous procedures, new ways to package Tcl extensions, Tcl-level custom channel types, file and line location information for each command, and more…
FME2008: New Screen Saver
Resident graphic designer Jara has just completed a screensaver with an FME2008 theme.
Download it from the fun stuff downloads page on fmepedia to find out what our wacky lizard is up to this time!
Linux FME Fixed Licensing: Important Notice
For reasons of reliability and security, the method of generating Registration Keys for Linux versions of FME has been changed for FME2009. This means that as of build 5574 of FME2009, any machine running a Linux version of FME with a fixed license will require a new license.
For more information see the bulletin on fmepedia.
Quick Info
- Reader/Writer and Function/Factory help files have been updated in build 5573
- We’re reviewing all keyboard shortcuts in Workbench - let me know if there are any you would specifically like to see.
- Beta builds of the FME 32-bit and 64-bit Linux versions are now automatically uploaded to our FTP site when available.
- The FSP (Forest Stewardship Plan) reader and writer has been updated to version 1.2 as of build 5562
- Safe announces partnership with WeoGeo to bring FME Server into the world of cloud computing
This Edition of the FME Evangelist…
…was written to the tune of “The Man In Me” by Bob Dylan.
http://www.youtube.com/watch?v=Y2s8_hCCHg4
This song is played in the intro of one of my favourite movies. Kudos if you know which one.
July 17th, 2008
RSS 2.0