Contents:
Raster Alpha Band Compositing
Concatenator Example
Google Earth Time Spans and Hurricanes
OS Grid Square Converter
This FME Evangelist highlights a number of interesting examples of FME use, including information on how to track hurricanes, how to map Scottish mountains, and why a demanding FME user is like a Buddhist ordering a hot dog.
Continue Reading September 13th, 2008
Contents:
1) Historical Markers
2) Complex Annotation Geometries
3) ArcGIS Image Server
4) Persistent SDE Connections
5) Text Size and Point Size
6) Quick Notes
Introduction
Hi FME’ers
In honour of the 2008 ESRI International User Conference, this post is a special on recent ESRI related updates in FME.
I think a key point of FME is how it easily integrates into spatial data workflows, simplifies processes, and helps people concentrate on using their data (rather than writing scripts or mucking about with text editors!). Therefore, for us at Safe, it’s vital that we keep our format support as up to date as possible; as even the smallest piece of unsupported functionality can force a user to take huge detours with their data.
That’s why it’s a real pleasure to be able to highlight improvements like these below. Browsing our user requests database I saw a number of funky ArcGIS functions - most of which I didn’t even realize existed - that we’d been asked to support. And here they are! Hopefully each one of these newly-implemented features will be a real time-saver.
The one caution is that, because these functions are newly added, they are very much still in beta form and subject to change and improvement as we get user feedback.
Regards,
Historical Markers
Historical markers are part of an archiving process introduced in ArcGIS v9.2 for ESRI’s SDE and Enterprise Geodatabase.
The basic idea is that - once activated - all database edits get stored in a separate archive class, with a date and time stamp. When viewing data you can - by specifying the appropriate time range - read the state of the spatial data as it existed at any particular moment in time. Additionally, an Historical Marker Manager lets you define moments in time with specific names that will obviously have greater meaning than dates (for example “As Designed” and “As Built”).
In FME2009 (as of beta build 5568) we now support querying data on the basis of these historical records. This is achieved by either selecting an historical marker on adding a reader, or entering a query into the appropriate reader parameter in an existing workspace.
Below: When adding a Geodatabase (SDE) reader to the workspace, the settings dialog includes this section. Here you can search out and specify a particular historical marker:
Below: Or you can choose to view data at a particular point in time by date:
Below: Once the reader is added to a workspace you can access historical data by setting the Archive Where Clause parameter:
Below: The Archive Where Clause is edited with this dialog. This example is a “range query” - ie the from and to dates are different.
According to the Readers and Writers Manual, a query can be a ‘moment’ query (the from and to dates are identical) or a ‘range’ query (the from and to dates are different). ESRI documentation tells me it’s also possible to query archives using attribute values (i.e. show me when <attr>=<value>) and Historical Markers, so - although I’ve no evidence - I believe this might also work in this FME parameter.
Complex Annotation Geometries
When, a few years ago, improvements were made to the FME geometry model, it enabled a whole deal of functionality which is still being unlocked. Support for complex ArcGIS annotation features is one of these.
By “complex” annotation geometries, we mean any text that is not a single piece of straight-line text, that originates at a specific point; i.e. text that is aligned to curves, arcs, polygons, multipoints, etc.
So, as of build 5555 we now support reading and writing of annotation with complex geometry.
Below: Bonus points for spotting the language of this curved annotation string.
Key things to remember:
Above: The Geometry Handling Mode must be Rich (not Classic)
ArcGIS Image Server
As of build 5550 Safe has added a new reader to FME: ArcGIS Image Server.
ArcGIS Image Server (or as of 9.3, ArcGIS Server Image) serves up file-based raster data to other applications. There are a number of applications with interfaces to use this data, including ArcGIS, AutoCAD and MicroStation; but the data can also be delivered as a web service, and this is where the FME reader might come into the picture.
Using FME on top of ArcGIS Server Image would firstly extend the format reach, secondly open up the data to all the raster transformation tools in FME, and thirdly allow everything to be integrated with all FME ETL functionality (for example, the ability to merge in vector datasources, live feeds, other raster data, text-based data etc etc) all “on-the-fly”.
The ArcGIS Server Image format is a reader only, since the product itself doesn’t have it’s own format (we read from a service). However for writing, it (the ArcGIS Server Image product) supports as source many of the same formats FME supports as a destination, plus I strongly suspect it would also accept a service as input, so you could create an “on-the-fly” FME writer using an FME Server service as an ArcGIS source.
NB: To get access to this format requires installation of the ArcGIS Image Server Client Core.
Persistent SDE Connections
Whenever FME connects to an SDE database it carries out the require processing and then shuts down the connection to save resources. However, when multiple tasks need to be carried out, creating a new connection each time is an expensive operation.
So that’s where a new keyword for FME2009 comes in. The PERSISTENT_CONNECTION keyword specifies whether to leave database connections open once their task is complete. That way other operations can use the same connection and avoid re-connection overheads.
Below: Persistent Connections are defined via a Reader Parameter:
This keyword isn’t intended for normal standalone translations, but for when the workspace is used in “super batch” mode or on FME Server; in these cases the reported improvement in connection time is a huge 50-60%. But just remember that an open connection is, effectively, an intentional one-time memory leak, so if you are not using SDE access in all translations it may not be worth the cost in system resources.
Text Size and Point Size
A curious discussion occurred here recently about whether “text size” and “font size” were the same thing, or different. It appears (from what I understand) that they are different; text size is the size of a text feature in ground (coordinate system) units, whereas font size is the size of the font to use in “point” units.
Therefore users were disappointed to find out that setting the format attribute geodb_text_size had no effect whatsoever on the size of the output font; and because font size is unrelated to geometry, the attribute geodb_text_ref_scale had no effect either.
The solution was for us to create a new format attribute: geodb_font_size
geodb_font_size defines the size of the font used to display the text string, in point units. However, there are some important points to note:
This fix was so important for a number of users we backported it to an updated FME2008 version. For more information see the Post-CD Fixes page on fmepedia.
Quick Notes
This Edition of the FME Evangelist…
…was written to the tune of “Greetings to the New Brunette” by Billy Bragg.
One of Britain’s best singer-songwriters. This is a slightly bizarre video, but you can’t argue with lines like:
“And if you hadn’t noticed yet, I’m more impressionable when my cement is wet.”
“How can you lie there and think of England, when you don’t even know who’s in the team?”
August 6th, 2008
Contents:
1) Raster To Vector Coercing
2) FFS Support for Raster Data
3) Visualizing Rasters
4) Persistent Cache and Raster
5) Transporting and CheckPointing Rasters
6) New Raster Formats
7) Raster Cell Origins
8) Raster Performance
9) Even more raster stuff…
Introduction
Hi FME’ers
This edition of the FME Evangelist is a special raster edition.
I know I wanted to vary topics in each post, to give something for everyone, but there’s so much raster info to go over that I can compile it into a single posting and still have enough left over to sprinkle into future posts. I have to say that our raster support is now very solid in FME, and well worth taking a look if you’ve not used it before. In particular I think being able to combine vector and raster data together, especially in FME Server applications, is going to be a big hit.
Regards,
Raster To Vector Coercing
Vectorizing raster data is a useful function that has just undergone a name change: RasterCellCoercer is the new name for the RasterToPointCoercer transformer. The reason it has been renamed is a very good one: the transformer now supports coercing raster data into polygons as well as points.
In the new ‘polygon’ mode a single polygon is output for each cell in the original raster. This results in a large number of polygon features - potentially useful in itself - but which you could dissolve together (based on the original cell value) to create a more manageable and realistic vector dataset.
In this example on fmepedia (screenshots below) a raster DEM dataset is converted into vector polygons using exactly this method.
FFS Support for Raster Data
I’m pleased to announce that as of FME 2009 beta build 5568 the FFS format is able to store raster features.
Rasters stored to an FFS file will have their data written to a corresponding .frs (FME Raster Store) file. One FRS file may hold the data for multiple raster features. FRS files hold up to 2GB of raster data; if the raster data surpasses the 2GB file size limit the data is automatically split across multiple files.
Above: A CDED raster dataset converted to FFS
Raster FFS datasets are used in FME (e.g. opened in the FME Viewer) by selecting the FFS file [though I tried opening the frs file and it did work - I just had to manually select the FFS format]. If you select an FFS file and the corresponding FRS file is missing then FME will just use the bounding box of the raster.
OK - writing raster to FFS might not seem like your idea of a good night out; however, it does integrate raster better with FME and open up some exciting benefits…
Visualizing Rasters
Did you see the previous FME Evangelist? If so, you may remember that:
“When you send data from Workbench to Viewer using a Visualizer transformer, what happens is that the data is simply written to a temporary FFS format dataset which is subsequently opened in the Viewer.”
In older versions of FME you’ve not been able to send raster data directly from Workbench to the Viewer - all you get is the dataset outline - because in between the data is written to FFS, and FFS doesn’t support raster.
But wait a moment! Hold the front page! FFS does now support raster.
So you are now able to direct raster data through the Visualizer transformer and get the correct view. One of those features a new user might say “yeah, so what?” about, but which an existing user will see as a great improvement.
Persistent Cache and Raster
Ever seen the FME Viewer option View > Options > View Persistent Cache and wondered what it does? Well turning it on causes all the data you open in the Viewer to be saved as FFS data in a temporary directory, so that every time you refresh the view or query the data you’re working on a local copy. That way you’re not having to re-read the source data, which can be a lengthy process if it is a web or database dataset.
Obviously this excludes raster, because raster can’t be saved as FFS. But wait! Hold the… er second page! FFS does now support raster.
So now using the Persistent Cache option in Viewer becomes a valuable time saver when inspecting raster data. The FME Viewer can even be shutdown and started at a later date and retain all the local copies of the data and thus the rapid access to the raster data.
The one word of warning is that this will create large amounts of data in the local temporary directory, for example, “C:/Documents and Settings/<username>/Local Settings/Temp/FMEUniversalViewer” - so make sure you have a lot of disk space, and try use your fastest disk.
Transporting and CheckPointing Rasters
Wait! Stop…. actually don’t stop the press on account of these functions; not many users are likely to have heard of these - let alone use them. But the ability to write raster to FFS also:
…and trust me, these are important updates, even if part of slightly less used features.
New Raster Formats
FME 2009 has new support for a couple of raster formats; the first of which may be of interest even to folk who are not generally raster users.
The Shuttle Radar Topography Mission (SRTM) Height format (short name SRTMHGT) is DEM data created by NASA. FME now has a reader for this format. A writer is not planned. The nice thing is that data in this format is freely available at NASA’s repository (though it wasn’t working when I tried it last)
Image Courtesy NASA/JPL-Caltech.
The data comes in two levels: SRTM-1 with data sampled at one arc-second intervals in latitude and longitude, and SRTM-3 sampled at three arc-seconds.
Datasets are divided into individual height files that each contain one by one degree latitude and longitude tiles. Height file names refer to the latitude and longitude of the lower left corner of the tile, for example, N37W105 has its lower left corner at 37 degrees North latitude and 105 degrees West longitude. Height files have the extension ‘.hgt’ and express the height values in meters. These files always have the LL84 coordinate system.
One gotcha is that FME is currently unable to read height files that do not conform to the specified file naming convention, since the file name is the only means of reading the proper referencing for the file.
ER Mapper ERS format (short name ERS) is the internal raster format used by ER Mapper. FME now has a reader AND writer for this format.
The ERS format consists of a header with a .ers extension and a separate raw binary data file that can have any extension. The header file is considered the dataset in FME and contains information about the contents of the binary data file.
Limitations are that the ERS writer currently cannot write nodata values, and that the reader does not support ERS files containing algorithms for image creation and manipulation.
Raster Cell Origins
As you know, a raster is made up of cells, each of which is a single point but which in fact covers a small area. What’s less known (perhaps) is that each cell has an origin point which identifies the representative location of each cell. The origin is an x/y pair between 0.0 and 1.0 inclusively. The default value in most cases is (0.5,0.5) indicating the center of the cell.
Cell origin is particularly important when the cell value is taken from a single position - when the value is the average value of the area the cell covers then cell origin is less important.
FME2009 includes new functionality to set a raster’s cell origin to specified X and Y values. The transformer is the RasterCellOriginSetter and the underlying function is @RasterCellOrigin
Why would you need to do this? Two reasons spring to mind.
Some formats of raster data don’t store cell origin; it is assumed to be a certain location (often upper-left corner, which is counter-intuitive for many map users). So this new tool allows a user to properly adjust data read from those formats, if they so desire.
The second use case is during raster-to-vector, or vector-to-raster, operations. In these scenarios you want to be sure that cells converted to points (or vice versa) end up in the correct location. If you don’t correctly set the cell origin you could find the resulting output is out by up to half the width of a cell.
If you don’t know what the current cell origin is, you can retrieve that information into attributes using the RasterPropertiesExtractor or by inspecting data using the Logger transformer.
Raster Performance
If you’ve not tried an FME2009 beta yet, it might be worth doing just for performance reasons.
Our raster team tells me that several common raster operations take less time to execute in FME2009.
Specifically, raster resampling and rotation should now be faster in all cases. The magnitude of improvement may vary from translation to translation, but these operations should be approximately 30-50% faster than in previous versions.
Raster reprojection benefits primarily when dealing with multi-band rasters. The common case of reprojecting a 3-band raster is now up to 50% faster.
Even more raster stuff…
If you have used FME2009 you may well have noticed the new RasterExpressionEvaluator transformer, and the new Raster clipping capabilities. Since they are so important I’ll talk about them another time: for now check out Dmitri’s RasterExpressionEvaluator examples for some absolutely stunning uses of this transformer, and try for yourself clipping raster features into non-rectangular shapes (very simple and effective).
This Edition of the FME Evangelist…
…was written to the tune of “Only You” by Yazoo.
I’ve been re-watching The Office (UK version) and the great Tim/Dawn romance. Here’s the end of the second series, and here’s the end of the following Christmas special where “Only You” is the background music.
Or, if you only want the music itself, here’s the original.
July 31st, 2008
Introduction
Hi folks. This is my first true posting to the FME Evangelist blog. In fact I’m on vacation at the moment, but can set a date for this to be published automatically. Ain’t technology great?!
Anyway, this issue has a nice mix of FME2009 updates, example workspaces, important notices, and an FME Server demo.
Cheers,
Contents
1) FME2009 Beta: Update
2) Logger Transformer: Feature Type in FFS Output
3) PointDisperser Customer Transformer
4) FME and ArcGIS v9.2 SP5
5) FME Server: Data Streaming Example
6) StringAttributeRounder Custom Transformer
7) VirtualEarthTiler: Spherical Mercator Coordinate System
8) Chopper Transformer: Overhaul for FME2009
FME2009 Beta: Update
Issue #12 noted that we were aware of a number of performance issues with the first FME2009 beta. I’m happy to say that all known issues are now reported as resolved and the current build (5557) back to normal.
Logger Transformer: Feature Type in FFS Output
The Logger transformer already had an example of its use on fmepedia, but with a number of updates since then (July 2006) it’s worth re-visiting this unassuming little transformer.
The main change you’ll find since that example was created is that features written to the Logger are now automatically written to a spatial Log file in FFS format. This is a great for visually inspecting features, rather than trying to interpret their representation in a plain text log file.
The other change is something new for FME2009; the ability to specify the feature type to which to write the spatial log features.
Previously the features sent to a Logger would go into a feature type named after the preceding transformer, for example Tester->Failed.
However, in FME2009 you are able to specify the feature type name, and so can be more descriptive – for example “Bad Address Data”, or “Unused Line Features”.
.
Above: The Feature Type to Log setting in the Logger transformer in FME2009
Visit the Logger example on fmepedia to see the updated version of the workspace.
Did You Know?
Did you know that FFS datasets come in both file and folder based flavours?
For me, the Logger transformer is particularly nice to use because multiple Loggers in the same workspace write their data to the same file-based dataset.
Compare this to the FFS format writer, where different feature types write to different FFS files within a folder (folder-based dataset).
There are benefits to both, but my preference is for a tidy, single-file dataset, so I choose the Logger every time to write my FFS files.
Below: The Logger transformer produces a file <workspaceName>_log.ffs…
…and that single FFS file contains two different feature types (Invalid Day and Invalid Month):
PointDisperser Custom Transformer
The PointDisperser custom transformer is a tool to create a form of structure suitable for thematic mapping. Where a number of points all belong to a certain parent location, the transformer can disperse them around the parent in a sort of radial diagram.
With a little more work the points and lines can be easily turned into coloured nodes and sectors.
These features look very effective when overlaid on a map background using some form of mapping or visualization tool.
In line with the concept of “eating one’s own dog food“, at Safe we use this tool for mapping sales figures, where the parent point represents a country or city, and each node represents an FME sale at that location.
No doubt you could envisage similar uses for your organization.
FME and ArcGIS v9.2 SP5
This is an important note for users of the Geodatabase readers and writers who upgrade their ArcGIS v9.2 installation to SP5.
FME 2007 and 2008 – which were both created before the release of SP5 – are both adversely affected by this upgrade. In particular you may receive a specific error message:
“The procedure entry point SgCodeFree could not be located in the dynamic link library sg.dll.”
Of course this will also impact any SpatialDirect and FME Server installations that use these readers and writers.
The quick-fix we have for this problem is to copy a number of files from the ArcGIS installation, and paste them into the FME installation folder, overwriting files of the same name.
The files are:
No problems have been reported using this workaround, but we don’t guarantee it because it’s a newer library than what FME was compiled against.
The long term fix is for us to issue updated versions of both FME2008 and FME2009, which we intend to do in the near future. If you want to be notified when this happens please email support@safe.com and quote PR#15697
NB: Only update the DLL files if you get the above error message. They should not need to be changed for any other reason.
FME Server: Data Streaming Example
Data streaming is a service provided by FME Server that delivers data right into an external application.
This example - which you may have seen as a demonstration before - is a very good illustration of that technique, and one of our favourites here at Safe.
Source data - in this case a live feed of Earthquake data from the USGS (GeoRSS format) - is translated into KML using a workspace that also restructures the data from simple points into extruded columns.
By uploading the workspace to FME Server the output data can be streamed directly into Google Earth – by anyone! The end user does not need FME to be able to visualize this data as all of the processing is done on an FME Server installation.
You can do this too! By simply clicking on this URL (or entering it into a web browser):
http://fmeserver.com:80/fmedatastreaming/translate/Samples/earthquakesextrusion.fmw
…you’ll make a call to an FME Server, which will carry out a translation, convert the data to KML, and stream it straight into Google Earth on your computer.
Above: The output viewed in Google Earth
The source data doesn’t need to be a live feed (as it is in this example). It can be any source data that you have on your system and that FME Server has access to - for example your spatial database (SDE, Oracle, SQL Server, etc). This means any person in your organisation can view your data - in Google Earth - immediately, and without you having to translate it for them.
Also, you can use more than one source dataset, in effect creating a Google Earth mashup with very little effort.
For example, you could merge a set of delivery routes with a live traffic alert feed (http://developer.yahoo.com/traffic/rss/V1/index.html) to show where a delivery schedule could be interrupted by traffic congestion.
StringAttributeRounder Custom Transformer
The AttributeRounder transformer is great for rounding the values of attributes (for example 123.456789 to 123.46) – but not so good when the attribute contains some sort of alphabetic character (for example 123.456789m). But rounding this form of text can be a requirement when the data is annotation read directly from a cartographic input.
The StringAttributeRounder is a custom transformer created in order to allow users to round off numeric attributes that are contained within a longer string.
The user can choose whether the text part of the string is at the start or end of the attribute.
For example:
Start Attribute: m234.5555
Text Part At: Start
Decimal Places: 2
Result Attribute: m234.56
Start Attribute: 234.5555m
Text Part At: End
Decimal Places: 2
Result Attribute: 234.56m
The transformer also handles <space> characters, for example:
Start Attribute: x y234.5555
Text Part At: Start
Decimal Places: 2
Result Attribute: x y234.56
If you want to use the transformer, or are interested to see how it works, then visit the StringAttributeRounder page on fmepedia. It requires FME2008, though there is also a version specifically for FME2007.
Above: In brief, the transformer strips off the text part of the string (from either the start or end), rounds off the remaining number, then glues it all back together with a Concatenator.
There is one specific best practice technique to look out for; the AttributeExpressionRemover transformer is used to clean up any temporary attributes created during the process, so the user is not distracted with extra data on the output.
VirtualEarthTiler: Spherical Mercator Coordinate System
A couple of users have experienced a problem with the VirtualEarthTiler in the last week, so I wanted to try to explain the cause (neither obvious nor simple), how to workaround it in FME2008, and how it has been resolved with FME2009.
The VirtualEarthTiler is used to turn a raster feature into a number of smaller tiles for use in Virtual Earth. It essentially does the same job as the MSR MapCruncher, but without the registration process.
The problem experienced was that the tiles that got written were not square; instead of being 256×256 they would be something like 400×256. To get square tiles required the ‘World File’ option to be turned on, posing the puzzling question “why should a world file make any difference to the size of the actual raster file itself?”
There are in fact two issues causing this to happen:
Issue One
Tiling data for VE (or GE) is more complicated than you’d think. Because source data is in plain LL and Virtual Earth uses a spherical coordinate system the output ends up with a latitude distortion. This distortion exhibits itself as non-square pixels in the raster output.
For example, in a test workspace I sent the VirtualEarthTiler output to a Logger and got this:
Number of Rows : 256
Number of Columns : 256
Cell Spacing : 0.0003433,0.00029677
As you can see, the cell spacing is different making these cells/pixels non-square in shape.
However the PNG format (as with many other raster formats) only has a single “field” in which to record cell spacing; i.e. it is assumed that X spacing is the same as Y. Therefore, because the PNG writer can’t write non-square cells, the data is resampled to make the cells square.
It’s this resampling that causes the overall size of the file to be changed from the expected 256×256.
Issue Two
FME resamples the data to create square cells because, if it did write non-square cells, how would another application know that there is odd spacing?
The answer is the world file.
That’s why writing a world (WLD) file makes a difference to the output.
FME is happy to write non-square cells provided there is a world file, because the world file tells other applications what the cell spacing is.
So, if you do write the world file then the cells will be non-rectangular, but you MUST keep the WLD file with the PNG else any application that reads the PNG will not know what the cell spacing is and so will not show the raster in its correct size and location.
FME2008
So, when you are using FME2008, you MUST have the ‘write world files’ setting turned on in your writer, and you MUST store the world files alongside the tiles for Virtual Earth to interpret the data correctly.
FME2009
Getting to the real point of this topic, FME2009 supports a new coordinate system called the Spherical Mercator, aka Web Mercator, EPSG:3785 or EPSG:900913*
Spherical Mercator is the coordinate system required for correct tiling of data. So convert your source raster to this coordinate system using the Reprojector, pipe it into the VirtualEarthTiler and you will get… tiles with perfectly square cells; without the need for world files either.
Problem solved.
* Get it? No neither did I for the longest time. I must be a n00b.
Chopper Transformer: Overhaul for FME2009
Many people who use the Chopper transformer do so to convert lines into points. However, did you know it can also be used to chop polygons into smaller area features? This might be useful where you have a maximum number of vertices requirement for your destination data.
Simply feed polygon features into the Chopper, and use a chop value of x (4 or greater), and the polygon will be split into a number of smaller area features. Each feature will have a maximum of x vertices.
Above: In this example the Chopper vertices parameter was set to 6, so no output polygon has more than 6 vertices.
This isn’t new functionality in itself, but what is new is that as of build 5556 in FME2009 beta the Chopper transformer has undergone a major improvement in the algorithm used to chop polygons.
The improvement results in:
For example:
Above: A polygon chopped using FME2008
Above: The same dataset chopped with FME2009
This Edition of the FME Evangelist…
…was written to the tune of “It doesn’t have to be this way” by the Blow Monkeys.
http://www.youtube.com/watch?v=co8P6fN732U
I don’t know how I came across this one, but it brought back some memories. 80’s music videos eh? Nothing like ‘em.
June 25th, 2008
RSS 2.0