The FME Evangelist

FME Evangelism Weekly Issue #11

Introduction

This “week’s” issue has important news for Linux, DGN and SQL Server users, plus a number of examples created using FME2008.

Also - this email coincides with the release of a new update to FME2008. This one is build 5204. You can get it from our ftp site at: ftp://ftp.safe.com/fme/beta/fme5204.exe

Although there aren’t any large-scale fixes - just a number of minor items that might affect a small number of users - there is a big enhancement with the ability to Write DGN Tags.

See item #2 below for more information on Tag writing, and check out fmepedia for the full list of updates: http://www.fmepedia.com/index.php/FME2008_Post-CD_Fixes

Cheers

Mark

Contents

1. The first 64-Bit FME!
2. DGN Tag writing in FME2008
3. FME and SQL Server Spatial May CTP (CTP-6 Refresh)
4. Checking for Self-Intersecting Lines
5. Weather Maps
6. VolumeCalculator
7. Neighbor Attribute Average
8. Articles on Safe and FME

1) The first 64-Bit FME!

In case you hadn’t noticed (not many people at Safe did!) the first 64-bit version of FME was released for FME2008. It is for Linux only (lucky penguin) and you can find it on the installation CD and on our FTP site.

Note, this is a true 64-bit version, not just one compatible with a 64-bit machine. The ordinary 32-bit version of FME for Linux (called linux-x86) will run on both x86 and x64 versions of Linux.

This 64-bit version will only run on Linux x64.

Also, where the 32-bit Linux FME is built for RedHat EL4 (x86), the 64-bit version is built for RedHat EL5 (x64).

Tux Picture (c) Larry Ewing

2) DGN Tag writing in FME2008

Anyone paying attention to the “What’s Great” page for FME2009 may have noticed that we have added tag writing functionality to that version of FME. For the non-CAD world, tags are a DGN format means of storing small amounts of attribute data.

On this topic there are two bits of excellent news for all you DGN users:

a) Tag writing has been back-ported into the updated version of FME2008 (build 5204). It was just so requested an item we had to do it.

This is what we’re calling ‘phase 1′ of this functionality. You need to create a list attribute for each tagset, each element in that list becoming a tag when you write the output. These would be the same tags you see when reading a dataset with tags.

There is also an option (keyword) to turn this functionality on and off.

There is an example of writing tags using FME2008 on fmepedia at:

http://www.fmepedia.com/index.php/Writing_DGN_Tags

b) For ‘phase 2′ of tag writing, FME2009 will allow you to add user attributes to your destination feature types (a new experience for you DGN users) and will (optionally) write these ‘attributes’ as tags on your DGN data.

More information on that functionality when a beta build is made available.

3) FME and SQL Server Spatial May CTP (CTP-6 Refresh)

If you read Spatial Ed (the blog page for Ed Katibah of Microsoft) you’ll already be aware that an upcoming release of SQL Server will swap the coordinate order for well-known text (WKT) and well-known binary (WKB) formats from latitude-longitude to the customer requested longitude-latitude.

So how will this affect FME users?

Well, if you upgrade SQL Server to the next CTP (for those not in the know CTP is a fancy acronym for beta version) then you should also upgrade your version of FME to FME2008 build 5200 or newer. You can read the list of updates made to that version on fmepedia and find the download on our ftp site.

Those sticking to the older CTP release of SQL Server should experience no problems, even if you still upgrade your FME. The new FME checks the version of SQL Server and acts accordingly.

4) Checking for Self-Intersecting Lines

As many of you will know, FME has a SelfIntersector transformer to remove self-intersections from lines.

Many users (and a few Safers) had always bemoaned the fact it intersects the data and won’t just check for self-intersections. The only way to do this was with a custom transformer like this one on fmepedia.

However, I’ve just found that the GeometryValidator transformer will test line features and flag as a failure any that self-intersect.

Who knew? Obviously not me.

Right: Yes, this line feature fails the “Simple” test and may therefore self-intersect!

5) Weather Maps

Dmitri Bagh - the “Q” of Safe Software - has been busy playing with web services again.

This time he’s dipped into a JSON service that returns information from weather stations within a user-defined bounding box.

It’s a useful example for anyone tasked with reading a JSON or GeoJSON dataset as it deals with the data using JSONExploder and JSONExtractor transformers. The bounding box parameters can - of course - be created using an existing dataset and the BoundsExtractor transformer, which makes this workspace flexible in terms of the area of interest.

You can find Dmitri’s workspace on fmepedia.

6) VolumeCalculator

Safe pro-services dude Dave Campanas recently came up with a useful custom transformer for calculating volume.

The volume is calculated using a set of 2.5D input features and a user-defined “base elevation”; so for calculating the difference between two surfaces you’d want to calculate their volumes separately (to the same base elevation) then simply subtract one from the other.

The transformer also accepts a boundary feature by which to define the area of interest.

http://www.fmepedia.com/index.php/VolumeCalculator

7) Neighbor Attribute Average

As for me, here’s my contribution to the fmepedia workspace pool: a Neighbor Attribute Average Calculator.

This workspace takes a set of adjoining polygons, examines a user-specified attribute and for each polygon calculates the average value of its neighbors.

One use - and I think this is what the requestor wanted it for - would be to identify spikes and wells in a DEM dataset. If the Z value of a polygon is greatly more or less than its neighbors average then it must be an erroneous value.

Do you have any other uses? Let us know.

http://www.fmepedia.com/index.php/Neighbor_Attribute_Average

8) Articles on Safe and FME

Check out the latest edition of GeoInformatics (p24) for an article by Safe president Don Murray on ‘The Growing Need for Spatial ETL’.

http://fluidbook.microdesign.nl/geoinformatics/03-2008/

Does the cover image look familiar?

If that weren’t enough, Dale also got his writer’s hat on and has two blog postings.

The first is about KML becoming an official OGC specification, and how its impact is comparable to the shockwaves felt when Wayne Gretzky went to play hockey for Los Angeles.

http://spatial-etl.blogspot.com/2008/05/kml-is-ogc-open-specification.html

The second is on the recent Where 2.0 show in San Francisco.

It tells how even if you know nano-formats, you might still not know Jack.

http://spatial-etl.blogspot.com/2008/05/safe-on-location-at-where-20.html

Star-Apic (an FME reseller) are holding a “Managing and Publishing Spatial Data” event at the Ordnance Survey offices in Southampton, UK.

Included in the events are a workshop on “Making the Most of FME”. See the Star-Apic web site for more details.

UK reseller Dotted Eyes will be running a FME event at their offices in Bromsgrove, Worcestershire on 18th June 2008. Email marketing@dottedeyes.com for more information.

Now that Celine Dion has finished her Las Vegas show, fellow-Canadian Dale Lutz is taking over with a three day stint at the Intergraph 2008 Conference. June 2nd-5th are the dates. Booth #805 and a 45 minute talk on Spatial ETL for GeoMedia.

For more upcoming events with Safe Software participation, see our web site: http://www.safe.com/aboutus/events/tradeshows.php

The FME Evangelist - this weekly in a more friendly (and email deliverable) form

FME User Central - a new access point for all FME technical resources…

…lots of things, but mostly the Guardian Football Weekly podcast.

In the latest issue the pod discuss John Terry, Avram Grant’s replacement and the play-off finals.

If football (soccer) isn’t your style then try the BBC’s Friday Night Comedy podcast instead.

May 26th, 2008

FME Evangelism Weekly Issue #10

Contents

1. FME 2008 Release Updates
2. AutoCAD Map 3D Object Data Reading Examples
3. Shape Datasets and the Age of Innocence
4. Google Spreadsheets Reader/Writer
5. Creating a Shape Index with FME
6. OIDs in PostGIS
7. Measurement Unit Converters

1) FME 2008 Release Updates

The first of the (previously mentioned) FME2008 Update builds is now available for download. Remember, these are minor fixes made available in lieu of an FME2009 beta.

You can download the first FME2008 update online (it’s now the default 2008 download) at:

http://www.safe.com/support/downloads.php

You can find a list of the changes in each particular build (this one is build 5200) on fmepedia at:

http://www.fmepedia.com/index.php/FME2008_Post-CD_Fixes

I’d suggest you read the list of updates before automatically installing this build, since only a very small minority of our users would be likely to find any benefit in the new build; but if you do have any questions don’t hesitate to contact support@safe.com

2) AutoCAD Map3D Object Data Reading - Example Workspaces

AutoCAD Map3D Object Data is a very flexible and open-ended format, which is great for you users but a challenge for us when creating the reader.

What we did was add one fairly unusual setting - a reading mode (right) - that affects how the source schema is displayed in your workspace.

Is essence you can get different views of the data model depending on what actions you want to carry out on the data.

To explain the different modes we created a set of example workspaces, based on the same dataset but each in a different reading mode.

These demonstrate the different results you can obtain with each mode and help explain when you would want to use them.

See the examples on fmepedia at: http://www.fmepedia.com/index.php/AutoCAD_Map_3D_Object_Data_Reading

3) Shape Datasets and the Age of Innocence

The Safe support team has just received their first Shape dataset more than 2GB in size. As Dale puts it, the age of innocence is over. The content below explains why.

NB: If your eyes glaze over at technical details then skip the bits marked with ###.

What is the Problem?

In theory a Shape dataset greater than 2GB in size is - if not impossible - certainly very implausible and difficult to handle. It should be noted that when I refer to a Shape dataset I mean the .shp part (the .dbf part has it’s own 2GB limit which is another problem by itself…)

### In technical terms this is because internal pointers between the index (shx) and spatial data (shp) are usually stored as signed 32-bit integers, so any size over 2^31 (2^31=2147483648 or 2GB) would need a larger number than that can store. So a computer application writing Shape data would find its counter overflows at (2^31)+1 and - if it didn’t crash at that point - start writing invalid pointers. ###

However, any software that writes a Shape file might ignore the problem, and just churn out bad indexes, which is how invalid datasets more than 2GB in size come to exist.

What does FME do?

When FME reaches the 2GB maximum it too ignores the problem (we were obviously still innocent when we implemented it) and continues to write data. However - while that leads to bad index pointers, they are bad in a way that is predicatable. So we made a change to our reader to handle this and now (in FME2008 build 5199 or greater) FME is able to read back any oversize datasets created by FME. We’d also be able to read Shape datasets from other naive applications that wrote data in the same way.

That sounds like a workaround. Is there anything else you can do?

Good question. ### Getting technical again, Shape internal pointers are actually measured in “words”. A “word” is a unit of data that depends on a computer’s architecture. For backwards compatibility reasons a 32-bit computer usually uses 16-bit words. So FME counts using a 32-bit signed integer but then divides by two to get the correct pointer. If you’re following along you might now say, “Ah! Since you divide everything by two, there’s no reaon why you can’t double the size of the pointers”; and you’d be right. We just can’t do the counting using a 32-bit integer, as an application usually would, because that would overflow at the 2GB mark. ###

So a future version of FME (FME2009, build 5525+) will be able to write up to 4GB Shape (again that’s .shp not .dbf) files. A fixed limit of 4GB will be hard-coded so we don’t get datasets any larger, which really would be invalid.

Not only will FME be able to read these 4GB datasets back, we believe the data is the format most likely to be readable by other applications.

Any problems to look out for?

Yes! Technically our theory is sound - and not contrary to the official Shape specification - just complex and relatively obscure. I don’t know of another application that is using this technique so we can’t guarantee that these applications (i.e. those still sweet and innocent) would be able to read our oversized datasets correctly, nor can we guarantee that we can read an oversized dataset created by another application (though even if they’ve created invalid data, if it’s in a predictable way, there’s a good chance we will).

Will a 64-bit computer support 4GB+ files?

Yes, but it would need a change in the Shape specification, along the same lines as recent changes in the TIFF format.

The TIFF format also had a size limit [### 4GB, so presumably they were using unsigned 32-bit pointers as opposed to signed with the 2GB-limited Shape ###] and so a new specification - BigTIFF - was created to allow Large File Support (LFS). Similar updates would need to occur to the Shape format to permit files greater than 4GB in size.

### So with (unsigned) 64-bit pointers we get 2^64 = 17 million TB - that’s Terabytes not Gigabytes folks, and even the most avid data collector is unlikely to have a Shape dataset that size. To put it in perspective, this is equivalent to 16,384 Petabytes (apparently Microsoft’s Virtual Earth imagery dataset is 14 petabytes in size) or 16 exabytes. According to wikipedia 16 exabytes of disk space would cost you $3 billion at today’s prices! ###

For the techno-confused amongst you (that includes me) here are some useful explanatory articles on wikipedia:

http://en.wikipedia.org/wiki/Large_file_support

http://en.wikipedia.org/wiki/Gigabyte

http://en.wikipedia.org/wiki/Word_%28computing%29

Now if you ever appear on Jeopardy you can feel confident in saying, “I’d like Shape datasets for one thousand please Alex”.

4) Google Spreadsheets Reader/Writer

If you weren’t aware, Google makes a Python API available for reading and writing Google data - ie data on any of the Google web services such as Documents, Picasa, YouTube (I didn’t even know that was a Google service), Calendar, etc

Pro-services dude Aaron has taken advantage of this - and FME’s ability to run Python scripts - to create an FME reader/writer (actually custom transformers) for Google Spreadsheets.

There are two versions of the Writer; full and fast.

The ‘full’ version writes features one at a time. It is slow but enables the use of multiple worksheets in a single spreadsheet.

The ‘fast’ version writes to a CSV file and uploads it as a new spreadsheet (but cannot write to an existing spreadsheet).

This is another great example of using FME to access web services.

For more details get the download from fmepedia at:

http://www.fmepedia.com/index.php/Google_Spreadsheets_Python_Read-Writelet

5) Creating a Shape Index with FME

A frequent question to the FME support team is, “How can I create a spatial index when I write a Shape dataset?” That’s something that previously wasn’t thought possible with FME, but now a combination of Python script and ArcObjects will let you do this.

How important would it be to have indexing when your Shape file is 4GB in size, eh?!

The key is that an ArcObjects Python script can index Shape files, and FME can run Python scripts passing into it such useful values as ‘destination dataset’. So by adding such a script as a Shutdown Python Script, it’s possible to create a spatial index for any dataset you have just written.

You can find an example workspace demonstrating this technique on fmepedia at:

http://www.fmepedia.com/index.php/Spatial_Indexes_for_ESRI_Shape_datasets

It’s simpler than it sounds because you just need to copy the Python script into the Shutdown script setting dialog (Navigator Pane > Workspace Settings > Advanced) and away you go. You can use the script in our workspace, but what’s really nice is that you can export a Python script from almost any tool in ArcToolbox and use that to do any number of things!

A few quick notes:

• You’ll need Python installed to run this (unlike TCL it isn’t included with FME). It can be found here.
• You’ll need ArcObjects installed (or at least the ArcGIS scripting library), which means you need an ESRI product such as ArcGIS.
• We didn’t get great results with Python v2.5 and ArcGIS v9.2, so we’d suggest using Python v2.4.
• A minor FME/Python fault means you can’t run a shutdown script without a startup script (at least not one that accesses parameters). So make sure to create a startup Python script in your workspace, even if it is empty.

No promises, but this functionality may even find its way into FME as a properly supported parameter! Remember, you heard it here first.

6) OIDs in PostGIS

Here’s a question and answer on the FMETalk user group that is probably worth passing on.

On 01/04/2008, Jeff wrote:

> Hi there,

> Could someone point out how to get FME to see the postgis oid column

> (created by FME when you choose ‘with oids’ in the format parameters)

> in the source features?

On 02/04/2008, Roland wrote:

> Morning Jeff,

> You can put in your own select statement in the feature properties.

> Under the ‘Parameters’ tab you’ll find inputs for WHERE clause and SELECT statement.

> In the SELECT box type in something like: select oid,* from <table_name>

> Attach an AttributeExposer with [an attribute] called ‘oid’ so that you can see it, and you’ll have OIDs (sounds painful).

Thanks to Jeff for posting the question and Roland for answering it; plus thanks to everyone else on the user group who participates in the FME community in this way.

7) Measurement Unit Converters

Safer Aaron Koning has really been busy (at salary review time - coincidence?!) - he’s also produced a series of custom transformers that translates length, area or volume attributes from one set of units into another.

This is a nice example of lots of things - using a Custom (Independent) published parameter, creating a custom transformer, use of the ValueMapper transformer, etc. I think I will add these to the FMEData sample dataset.

It’s not just yards or metres that it will convert either; there are lots of different units such as miles, chains, hectares, acres and six different types of foot measurement!

So if you ever wanted to convert your LengthCalculator results from your current units into twips, light-years or angstroms, here’s your chance!

Check out fmepedia to get these great custom transformers:

http://fmepedia.com/index.php/Measurement_Unit_Converters

• The Spring 2008 edition of the FME Insider newsletter is now available. Apologies to the southern hemisphere for whom it is anything but Spring!
• An Austrian FME user conference will take place in Vienna on 20th May 2008. Click here for more details.
• Interested in Mastermap and/or SQL Server Spatial? See this blog posting for a great example that uses both.
• A month or two old, but here’s a blog posting about Don Murray’s visit to a conference on Geospatial Data Standards.

Gerry Rafferty’s superb album North and South.

It’s my favourite of his albums, but relatively unknown. The only track I can find on YouTube is “Nothin’ Ever Happens Down Here“.

May 5th, 2008