Contents:
1) Spatial Data Validation…
2) …and the FME Server
3) NeighborFinder Update
4) Oracle Connections
This FME Evangelist post includes information on how to validate spatial data, how to apply that validation to submissions using FME Server, how to speed up nearest neighbo(u)r operations, and how to resolve Oracle connection problems.
Continue Reading August 12th, 2008
Contents:
1) Historical Markers
2) Complex Annotation Geometries
3) ArcGIS Image Server
4) Persistent SDE Connections
5) Text Size and Point Size
6) Quick Notes
Introduction
Hi FME’ers
In honour of the 2008 ESRI International User Conference, this post is a special on recent ESRI related updates in FME.
I think a key point of FME is how it easily integrates into spatial data workflows, simplifies processes, and helps people concentrate on using their data (rather than writing scripts or mucking about with text editors!). Therefore, for us at Safe, it’s vital that we keep our format support as up to date as possible; as even the smallest piece of unsupported functionality can force a user to take huge detours with their data.
That’s why it’s a real pleasure to be able to highlight improvements like these below. Browsing our user requests database I saw a number of funky ArcGIS functions - most of which I didn’t even realize existed - that we’d been asked to support. And here they are! Hopefully each one of these newly-implemented features will be a real time-saver.
The one caution is that, because these functions are newly added, they are very much still in beta form and subject to change and improvement as we get user feedback.
Regards,
Historical Markers
Historical markers are part of an archiving process introduced in ArcGIS v9.2 for ESRI’s SDE and Enterprise Geodatabase.
The basic idea is that - once activated - all database edits get stored in a separate archive class, with a date and time stamp. When viewing data you can - by specifying the appropriate time range - read the state of the spatial data as it existed at any particular moment in time. Additionally, an Historical Marker Manager lets you define moments in time with specific names that will obviously have greater meaning than dates (for example “As Designed” and “As Built”).
In FME2009 (as of beta build 5568) we now support querying data on the basis of these historical records. This is achieved by either selecting an historical marker on adding a reader, or entering a query into the appropriate reader parameter in an existing workspace.
Below: When adding a Geodatabase (SDE) reader to the workspace, the settings dialog includes this section. Here you can search out and specify a particular historical marker:
Below: Or you can choose to view data at a particular point in time by date:
Below: Once the reader is added to a workspace you can access historical data by setting the Archive Where Clause parameter:
Below: The Archive Where Clause is edited with this dialog. This example is a “range query” - ie the from and to dates are different.
According to the Readers and Writers Manual, a query can be a ‘moment’ query (the from and to dates are identical) or a ‘range’ query (the from and to dates are different). ESRI documentation tells me it’s also possible to query archives using attribute values (i.e. show me when <attr>=<value>) and Historical Markers, so - although I’ve no evidence - I believe this might also work in this FME parameter.
Complex Annotation Geometries
When, a few years ago, improvements were made to the FME geometry model, it enabled a whole deal of functionality which is still being unlocked. Support for complex ArcGIS annotation features is one of these.
By “complex” annotation geometries, we mean any text that is not a single piece of straight-line text, that originates at a specific point; i.e. text that is aligned to curves, arcs, polygons, multipoints, etc.
So, as of build 5555 we now support reading and writing of annotation with complex geometry.
Below: Bonus points for spotting the language of this curved annotation string.
Key things to remember:
Above: The Geometry Handling Mode must be Rich (not Classic)
ArcGIS Image Server
As of build 5550 Safe has added a new reader to FME: ArcGIS Image Server.
ArcGIS Image Server (or as of 9.3, ArcGIS Server Image) serves up file-based raster data to other applications. There are a number of applications with interfaces to use this data, including ArcGIS, AutoCAD and MicroStation; but the data can also be delivered as a web service, and this is where the FME reader might come into the picture.
Using FME on top of ArcGIS Server Image would firstly extend the format reach, secondly open up the data to all the raster transformation tools in FME, and thirdly allow everything to be integrated with all FME ETL functionality (for example, the ability to merge in vector datasources, live feeds, other raster data, text-based data etc etc) all “on-the-fly”.
The ArcGIS Server Image format is a reader only, since the product itself doesn’t have it’s own format (we read from a service). However for writing, it (the ArcGIS Server Image product) supports as source many of the same formats FME supports as a destination, plus I strongly suspect it would also accept a service as input, so you could create an “on-the-fly” FME writer using an FME Server service as an ArcGIS source.
NB: To get access to this format requires installation of the ArcGIS Image Server Client Core.
Persistent SDE Connections
Whenever FME connects to an SDE database it carries out the require processing and then shuts down the connection to save resources. However, when multiple tasks need to be carried out, creating a new connection each time is an expensive operation.
So that’s where a new keyword for FME2009 comes in. The PERSISTENT_CONNECTION keyword specifies whether to leave database connections open once their task is complete. That way other operations can use the same connection and avoid re-connection overheads.
Below: Persistent Connections are defined via a Reader Parameter:
This keyword isn’t intended for normal standalone translations, but for when the workspace is used in “super batch” mode or on FME Server; in these cases the reported improvement in connection time is a huge 50-60%. But just remember that an open connection is, effectively, an intentional one-time memory leak, so if you are not using SDE access in all translations it may not be worth the cost in system resources.
Text Size and Point Size
A curious discussion occurred here recently about whether “text size” and “font size” were the same thing, or different. It appears (from what I understand) that they are different; text size is the size of a text feature in ground (coordinate system) units, whereas font size is the size of the font to use in “point” units.
Therefore users were disappointed to find out that setting the format attribute geodb_text_size had no effect whatsoever on the size of the output font; and because font size is unrelated to geometry, the attribute geodb_text_ref_scale had no effect either.
The solution was for us to create a new format attribute: geodb_font_size
geodb_font_size defines the size of the font used to display the text string, in point units. However, there are some important points to note:
This fix was so important for a number of users we backported it to an updated FME2008 version. For more information see the Post-CD Fixes page on fmepedia.
Quick Notes
This Edition of the FME Evangelist…
…was written to the tune of “Greetings to the New Brunette” by Billy Bragg.
One of Britain’s best singer-songwriters. This is a slightly bizarre video, but you can’t argue with lines like:
“And if you hadn’t noticed yet, I’m more impressionable when my cement is wet.”
“How can you lie there and think of England, when you don’t even know who’s in the team?”
August 6th, 2008
Introduction
Today’s email is an ad-hoc info session on the FME2009 beta. Expect more follow ups in the near future.
Cheers,
Mark
Contents
1) First FME2009 Beta now Released!
2) New GUI Interface
3) Transformer Changes
4) Performance Issues
5) Cool Format Updates
6) Cool Functionality Updates
7) The Complete What’s Great
1) First FME2009 Beta now Released!
Yes. You heard it here first.
The first FME2009 beta build is going to be released in the next couple of hours, if not already.
Hence the need to brief you all on the changes you will find.
As usual you can obtain it from our web site downloads page, or directly from our ftp site.
As with any beta version, be aware that it is subject to change and the functionality might not be the final version.
We don’t really recommend using a beta for production work unless there’s a specific bug fix that you need (and since most of those have been ported to an updated 2008 build, there’s even less excuse).
The biggest thing that you’ll notice about this version of FME is a completely overhauled user interface for settings boxes.
You may have noticed a couple of these sneak into FME2008 (for example the ‘Add Published Parameter’ dialog), but now the entire bunch of settings dialogs have been overhauled to use this design.
The Bufferer transformer is a good example. The new dialog looks like this (below):
Notice that required fields are all highlighted in yellow, to show you where to fill in compulsory values, and that you are also prompted on which type of data to enter (eg “Floating Point Value”, “String or Attribute Name”, etc).
It’s not obvious from a static image, but these dialogs are also capable of being resized, which is something you couldn’t do with the old dialogs - for example (below):
This update applies to both transformer settings and reader/writer settings dialogs too.
From our point of view, the really nice thing about this upgrade is that we now have a whole lot more options for advanced GUI features (for example greying out one setting based on the value of another) which will eventually lead to a more logical interface. Another benefit is that these dialogs are compatible with Linux operating systems, bringing us one step closer to relealizing a full FME version for Linux.
If you are looking for the LineGeneralizer or AreaGeneralizer transformer, you’ll find that we’ve combined them into a single transformer called the Generalizer.
Likewise, if you are looking for the Rasterizer transformer, be aware that it’s been split up into the ImageRasterizer and NumericRasterizer.
You need to be aware that our in-house test suite has identified a number of performance issues that are still being addressed.
a) Six of our tests required a reduced value for the “Max Features in Memory” setting before they would complete. That’s six tests out of several thousand, but it’s possible that you may need to adjust this setting to get an existing workspace to complete.
b) Performance (speed) suffered when we moved to a new programming environment. We know why and it’s being worked on, but for the time being some translations may run up to 50% slower.
c) We appear to take a performance hit when reading/writing networked datasets (ie files that are not on the local filesystem). Again, we’re looking into the causes and will fix it asap.
There have been a number of cool updates to formats that are worth mentioning here in brief:
a) DGN Tag Writing
Here’s something you’ll not have seen before (below). A DGN destination feature type with attributes on it.
Workspaces writing to DGN will now generate an attribute schema, and the attributes will be written to tags.
Left: Tag writing is turned on/off using a setting in the navigation pane. Default is currently on.
b) MapInfo MDB/XLS Support
The MapInfo MITAB reader now supports MapInfo TAB datasets where attributes are stored in either an MDB or XLS spreadsheet.
c) Adobe PDF Updates
PDF output now permits control over background colour, page size and ambient/specular colours.
d) Database SQL Statements
Database formats that permit the concept now support multiple before/after SQL statements. A semicolon is used for the delimiter between statements.
e) BigTIFF Support
The GeoTIFF writer now supports the ability to write BigTIFF datasets.
f) Autocad DWG MPolygon support
The DWG reader now supports MPolygon entities.
There have also been a number of cool updates to FME functionality that are worth mentioning here in brief:
a) Arc Stroking
The stroking of arcs (well actually any curve) can now be controlled by a new directive FME_STROKE_MAX_DEVIATION which defines the maximum distance between the arc and the stroked chord. This is exposed in the Workbench GUI and in various transformers. It’s key use is to control internal operations - for example when writing to a non arc-supporting format.
b) Raster Clipping
The Raster Clipper now supports non-rectangular clipping. This is way way way cool.
c) Raster Expression Evaluator
This new transformer lets you carry out algebra type functions on raster cells.
Dmitri has a great page full of examples.
Don’t forget that the full list of Great items can be found on fmepedia:
Also check out Dmitri’s fmepedia pages for lots of examples.
http://www.fmepedia.com/index.php/Category:Dmitris_Raster_Studio
http://www.fmepedia.com/index.php/Category:Dmitris_3D_Place
http://www.fmepedia.com/index.php/Category:Web_Services
The evangelism email will look at all of these new functions and formats in closer detail in the near future.
Don’t forget to email us feedback and suggestions for any of this new functionality.
June 2nd, 2008
Introduction
This “week’s” issue has important news for Linux, DGN and SQL Server users, plus a number of examples created using FME2008.
Also - this email coincides with the release of a new update to FME2008. This one is build 5204. You can get it from our ftp site at: ftp://ftp.safe.com/fme/beta/fme5204.exe
Although there aren’t any large-scale fixes - just a number of minor items that might affect a small number of users - there is a big enhancement with the ability to Write DGN Tags.
See item #2 below for more information on Tag writing, and check out fmepedia for the full list of updates: http://www.fmepedia.com/index.php/FME2008_Post-CD_Fixes
Cheers
Mark
Contents
1. The first 64-Bit FME!
2. DGN Tag writing in FME2008
3. FME and SQL Server Spatial May CTP (CTP-6 Refresh)
4. Checking for Self-Intersecting Lines
5. Weather Maps
6. VolumeCalculator
7. Neighbor Attribute Average
8. Articles on Safe and FME
In case you hadn’t noticed (not many people at Safe did!) the first 64-bit version of FME was released for FME2008. It is for Linux only (lucky penguin) and you can find it on the installation CD and on our FTP site.
Note, this is a true 64-bit version, not just one compatible with a 64-bit machine. The ordinary 32-bit version of FME for Linux (called linux-x86) will run on both x86 and x64 versions of Linux.
This 64-bit version will only run on Linux x64.
Also, where the 32-bit Linux FME is built for RedHat EL4 (x86), the 64-bit version is built for RedHat EL5 (x64).
Tux Picture (c) Larry Ewing
Anyone paying attention to the “What’s Great” page for FME2009 may have noticed that we have added tag writing functionality to that version of FME. For the non-CAD world, tags are a DGN format means of storing small amounts of attribute data.
On this topic there are two bits of excellent news for all you DGN users:
a) Tag writing has been back-ported into the updated version of FME2008 (build 5204). It was just so requested an item we had to do it.
This is what we’re calling ‘phase 1′ of this functionality. You need to create a list attribute for each tagset, each element in that list becoming a tag when you write the output. These would be the same tags you see when reading a dataset with tags.
There is also an option (keyword) to turn this functionality on and off.
There is an example of writing tags using FME2008 on fmepedia at:
http://www.fmepedia.com/index.php/Writing_DGN_Tags
b) For ‘phase 2′ of tag writing, FME2009 will allow you to add user attributes to your destination feature types (a new experience for you DGN users) and will (optionally) write these ‘attributes’ as tags on your DGN data.
More information on that functionality when a beta build is made available.
3) FME and SQL Server Spatial May CTP (CTP-6 Refresh)
If you read Spatial Ed (the blog page for Ed Katibah of Microsoft) you’ll already be aware that an upcoming release of SQL Server will swap the coordinate order for well-known text (WKT) and well-known binary (WKB) formats from latitude-longitude to the customer requested longitude-latitude.
So how will this affect FME users?
Well, if you upgrade SQL Server to the next CTP (for those not in the know CTP is a fancy acronym for beta version) then you should also upgrade your version of FME to FME2008 build 5200 or newer. You can read the list of updates made to that version on fmepedia and find the download on our ftp site.
Those sticking to the older CTP release of SQL Server should experience no problems, even if you still upgrade your FME. The new FME checks the version of SQL Server and acts accordingly.
4) Checking for Self-Intersecting Lines
As many of you will know, FME has a SelfIntersector transformer to remove self-intersections from lines.
Many users (and a few Safers) had always bemoaned the fact it intersects the data and won’t just check for self-intersections. The only way to do this was with a custom transformer like this one on fmepedia.
However, I’ve just found that the GeometryValidator transformer will test line features and flag as a failure any that self-intersect.
Who knew? Obviously not me.
Right: Yes, this line feature fails the “Simple” test and may therefore self-intersect!
Dmitri Bagh - the “Q” of Safe Software - has been busy playing with web services again.
This time he’s dipped into a JSON service that returns information from weather stations within a user-defined bounding box.
It’s a useful example for anyone tasked with reading a JSON or GeoJSON dataset as it deals with the data using JSONExploder and JSONExtractor transformers. The bounding box parameters can - of course - be created using an existing dataset and the BoundsExtractor transformer, which makes this workspace flexible in terms of the area of interest.
You can find Dmitri’s workspace on fmepedia.
Safe pro-services dude Dave Campanas recently came up with a useful custom transformer for calculating volume.
The volume is calculated using a set of 2.5D input features and a user-defined “base elevation”; so for calculating the difference between two surfaces you’d want to calculate their volumes separately (to the same base elevation) then simply subtract one from the other.
The transformer also accepts a boundary feature by which to define the area of interest.
http://www.fmepedia.com/index.php/VolumeCalculator
As for me, here’s my contribution to the fmepedia workspace pool: a Neighbor Attribute Average Calculator.
This workspace takes a set of adjoining polygons, examines a user-specified attribute and for each polygon calculates the average value of its neighbors.
One use - and I think this is what the requestor wanted it for - would be to identify spikes and wells in a DEM dataset. If the Z value of a polygon is greatly more or less than its neighbors average then it must be an erroneous value.
Do you have any other uses? Let us know.
http://www.fmepedia.com/index.php/Neighbor_Attribute_Average
Check out the latest edition of GeoInformatics (p24) for an article by Safe president Don Murray on ‘The Growing Need for Spatial ETL’.
http://fluidbook.microdesign.nl/geoinformatics/03-2008/
Does the cover image look familiar?
If that weren’t enough, Dale also got his writer’s hat on and has two blog postings.
The first is about KML becoming an official OGC specification, and how its impact is comparable to the shockwaves felt when Wayne Gretzky went to play hockey for Los Angeles.
http://spatial-etl.blogspot.com/2008/05/kml-is-ogc-open-specification.html
The second is on the recent Where 2.0 show in San Francisco.
It tells how even if you know nano-formats, you might still not know Jack.
http://spatial-etl.blogspot.com/2008/05/safe-on-location-at-where-20.html
Star-Apic (an FME reseller) are holding a “Managing and Publishing Spatial Data” event at the Ordnance Survey offices in Southampton, UK.
Included in the events are a workshop on “Making the Most of FME”. See the Star-Apic web site for more details.
UK reseller Dotted Eyes will be running a FME event at their offices in Bromsgrove, Worcestershire on 18th June 2008. Email marketing@dottedeyes.com for more information.
Now that Celine Dion has finished her Las Vegas show, fellow-Canadian Dale Lutz is taking over with a three day stint at the Intergraph 2008 Conference. June 2nd-5th are the dates. Booth #805 and a 45 minute talk on Spatial ETL for GeoMedia.
For more upcoming events with Safe Software participation, see our web site: http://www.safe.com/aboutus/events/tradeshows.php
The FME Evangelist - this weekly in a more friendly (and email deliverable) form
FME User Central - a new access point for all FME technical resources…
…lots of things, but mostly the Guardian Football Weekly podcast.
In the latest issue the pod discuss John Terry, Avram Grant’s replacement and the play-off finals.
If football (soccer) isn’t your style then try the BBC’s Friday Night Comedy podcast instead.
May 26th, 2008
RSS 2.0