Posts filed under 'Training'
Contents:
A New Chapter in the History of Workbench: Quick Connect,
Your Spy at Safe: New Training Facility,
Anatomy of an FME Project: Parts V and VI,
Concatenator Tip,
Shutdown Scripts and Log Files
This FME Evangelist reports on an important update to FME Workbench, gives insider news on Safe Software’s new training facility, details a couple of useful updates for handling log files in Tcl and Python, plus continues the series “Anatomy of an FME Project” by fetching maps from a web server and outputting them inside a custom HTML template.
Continue Reading
November 10th, 2008
Contents:
FME/Smallworld Background,
FME and the Smallworld SIAS,
Behind the Scenes,
Technical Details of a Smallworld Workspace,
Data Streaming and Linking,
What’s New in the FME SpatialBiz Plug-In,
Smallworld 4.1 Support,
Training Offer.
This FME Evangelist is a special edition for the 2008 Smallworld Americas Users Conference. It details a particular scenario that we will be demonstrating at the conference: how to integrate FME Server with a web mapping tool such as SIAS (Smallworld Internet Application Server). It has good information for anyone new to FME, plus news about recent FME-Smallworld enhancements, and mentions techniques useful to most users.
Continue Reading
September 27th, 2008
Contents:
1) Historical Markers
2) Complex Annotation Geometries
3) ArcGIS Image Server
4) Persistent SDE Connections
5) Text Size and Point Size
6) Quick Notes
Introduction
Hi FME’ers
In honour of the 2008 ESRI International User Conference, this post is a special on recent ESRI related updates in FME.
I think a key point of FME is how it easily integrates into spatial data workflows, simplifies processes, and helps people concentrate on using their data (rather than writing scripts or mucking about with text editors!). Therefore, for us at Safe, it’s vital that we keep our format support as up to date as possible; as even the smallest piece of unsupported functionality can force a user to take huge detours with their data.
That’s why it’s a real pleasure to be able to highlight improvements like these below. Browsing our user requests database I saw a number of funky ArcGIS functions - most of which I didn’t even realize existed - that we’d been asked to support. And here they are! Hopefully each one of these newly-implemented features will be a real time-saver.
The one caution is that, because these functions are newly added, they are very much still in beta form and subject to change and improvement as we get user feedback.
Regards,
Historical Markers
Historical markers are part of an archiving process introduced in ArcGIS v9.2 for ESRI’s SDE and Enterprise Geodatabase.
The basic idea is that - once activated - all database edits get stored in a separate archive class, with a date and time stamp. When viewing data you can - by specifying the appropriate time range - read the state of the spatial data as it existed at any particular moment in time. Additionally, an Historical Marker Manager lets you define moments in time with specific names that will obviously have greater meaning than dates (for example “As Designed” and “As Built”).
In FME2009 (as of beta build 5568) we now support querying data on the basis of these historical records. This is achieved by either selecting an historical marker on adding a reader, or entering a query into the appropriate reader parameter in an existing workspace.
Below: When adding a Geodatabase (SDE) reader to the workspace, the settings dialog includes this section. Here you can search out and specify a particular historical marker:
Below: Or you can choose to view data at a particular point in time by date:
Below: Once the reader is added to a workspace you can access historical data by setting the Archive Where Clause parameter:
Below: The Archive Where Clause is edited with this dialog. This example is a “range query” - ie the from and to dates are different.
According to the Readers and Writers Manual, a query can be a ‘moment’ query (the from and to dates are identical) or a ‘range’ query (the from and to dates are different). ESRI documentation tells me it’s also possible to query archives using attribute values (i.e. show me when <attr>=<value>) and Historical Markers, so - although I’ve no evidence - I believe this might also work in this FME parameter.
Complex Annotation Geometries
When, a few years ago, improvements were made to the FME geometry model, it enabled a whole deal of functionality which is still being unlocked. Support for complex ArcGIS annotation features is one of these.
By “complex” annotation geometries, we mean any text that is not a single piece of straight-line text, that originates at a specific point; i.e. text that is aligned to curves, arcs, polygons, multipoints, etc.
So, as of build 5555 we now support reading and writing of annotation with complex geometry.
Below: Bonus points for spotting the language of this curved annotation string.
Key things to remember:
- You MUST have the Geometry Handling parameter (shown below) set to “Rich”. You won’t be able to write complex annotation geometries (and maybe not read) if it is set to “Classic”.
- The data inspected in the FME Viewer will not show curved text - that’s because it isn’t supported (i.e. the Viewer is only meant for data inspection, not visualization)
- Splines and curves will be stroked into line features; we do hope to properly support these geometries in the near future.
Above: The Geometry Handling Mode must be Rich (not Classic)
ArcGIS Image Server
As of build 5550 Safe has added a new reader to FME: ArcGIS Image Server.
ArcGIS Image Server (or as of 9.3, ArcGIS Server Image) serves up file-based raster data to other applications. There are a number of applications with interfaces to use this data, including ArcGIS, AutoCAD and MicroStation; but the data can also be delivered as a web service, and this is where the FME reader might come into the picture.
Using FME on top of ArcGIS Server Image would firstly extend the format reach, secondly open up the data to all the raster transformation tools in FME, and thirdly allow everything to be integrated with all FME ETL functionality (for example, the ability to merge in vector datasources, live feeds, other raster data, text-based data etc etc) all “on-the-fly”.
The ArcGIS Server Image format is a reader only, since the product itself doesn’t have it’s own format (we read from a service). However for writing, it (the ArcGIS Server Image product) supports as source many of the same formats FME supports as a destination, plus I strongly suspect it would also accept a service as input, so you could create an “on-the-fly” FME writer using an FME Server service as an ArcGIS source.
NB: To get access to this format requires installation of the ArcGIS Image Server Client Core.
Persistent SDE Connections
Whenever FME connects to an SDE database it carries out the require processing and then shuts down the connection to save resources. However, when multiple tasks need to be carried out, creating a new connection each time is an expensive operation.
So that’s where a new keyword for FME2009 comes in. The PERSISTENT_CONNECTION keyword specifies whether to leave database connections open once their task is complete. That way other operations can use the same connection and avoid re-connection overheads.
Below: Persistent Connections are defined via a Reader Parameter:
This keyword isn’t intended for normal standalone translations, but for when the workspace is used in “super batch” mode or on FME Server; in these cases the reported improvement in connection time is a huge 50-60%. But just remember that an open connection is, effectively, an intentional one-time memory leak, so if you are not using SDE access in all translations it may not be worth the cost in system resources.
Text Size and Point Size
A curious discussion occurred here recently about whether “text size” and “font size” were the same thing, or different. It appears (from what I understand) that they are different; text size is the size of a text feature in ground (coordinate system) units, whereas font size is the size of the font to use in “point” units.
Therefore users were disappointed to find out that setting the format attribute geodb_text_size had no effect whatsoever on the size of the output font; and because font size is unrelated to geometry, the attribute geodb_text_ref_scale had no effect either.
The solution was for us to create a new format attribute: geodb_font_size
geodb_font_size defines the size of the font used to display the text string, in point units. However, there are some important points to note:
- Once more, you MUST have the Geometry Handling parameter set to “Rich”. The attribute would be ignored if the geometry handling mode is set to “Classic”.
- For the sake of backwards compatibility, geodb_text_size will overrule geodb_font_size if both are set. So remove any reference to geodb_text_size if you are wanting to set a text font size.
This fix was so important for a number of users we backported it to an updated FME2008 version. For more information see the Post-CD Fixes page on fmepedia.
Quick Notes
- The “Split Multi Part Annotations” option available on the Geodatabase_MDB reader has also been exposed on the SDE Geodatabase and File Geodatabase readers as of build 5564.
- An equivalent format attribute for the ArcGIS parameter “Character Width” has been created as of build 5531. It is called geodb_text_character_width
- The personal geodatabase metadata attribute, fme_num_entries, is also exposed in the file geodatabase reader/writer as of build 5529
- Did you catch Don Murray and Mark Stoakes presenting a pre-conference seminar on Data Interoperability? If not, don’t worry. Safe-hosted ArcGIS Data Interoperability Training will take place in Denver, Houston and Washington DC in the coming months. See the Safe training calendar for more information.
This Edition of the FME Evangelist…
…was written to the tune of “Greetings to the New Brunette” by Billy Bragg.
One of Britain’s best singer-songwriters. This is a slightly bizarre video, but you can’t argue with lines like:
“And if you hadn’t noticed yet, I’m more impressionable when my cement is wet.”
“How can you lie there and think of England, when you don’t even know who’s in the team?”
August 6th, 2008
Introduction
Back after a little break… well I did say this would be weekly-ish.
Again this issue is more related to use of FME than new development, since we’re still in the final phases of the 08 release.
Contents
- Training Resources for FME 2008
- New Format: Oracle GeoRaster Reader
- RasterRotationApplier Transformer and @ApplyRasterRotation function
- Random Number Generation via a Web Service
- New Format: Directory and File Pathnames Reader
- FME Group on Facebook
- Data Cleanup and Validation
1) Training Resources for FME 2008
The training materials for FME 2008 are complete (phew!) and will be debuted at the FME User Conference training courses.
In order to achieve our two main aims - training as many users as possible, and providing value for certified trainers - for 2008 we have a number of different training packages aimed at different levels of skill and expertise.
- At the top of the pile are certified trainers. They receive the full set of materials including advanced modules, instructor notes and training certificates, plus receive other certification benefits such as exposure on our web site.
- One step down, users who have applied to our certification program receive more a set of detailed materials to help them bring their training skills up to a certifiable level.
- At the next level, resellers and educational establishments can download a set of basic materials for general educational use by non-certified trainers.
- At the lowest level, users are able to download a set of basic self-study materials as PDFs.
The Online Learning page of the Safe web site will connect you to all public training resources, including the FME Tutorial and the Self-Study materials (nb: the 2008 materials are still awaiting uploading), while the other specialist levels are available either on demand or through the reseller and certification portals.
2) New Format: Oracle GeoRaster Reader
The Oracle GeoRaster reader is a new format for FME2008.
This reader nicely complements the Oracle GeoRaster writer that already exists, since raster data can now be both imported and exported from an Oracle database using FME.
This is an entirely separate reader to the Oracle vector reader, but is documented within the same section of the Readers and Writers Manual.
The table pickers for our Oracle readers are able to tell between vector and raster tables, and will only show the table when it contains the relevant type of data.
The latest builds also include the ability to choose a specific pyramid to read.
The Oracle GeoRaster reader supports most of the numeric and color interpretation types on both bands and palettes supported by FME. It does not support some of Oracle’s more advanced types such as 128 bit Complex Numbers. A search envelope may also be applied to the reader to restrict data retrieval to an area of interest.
The reader supports SRS georeferencing, including rotation of raster features.
3) RasterRotationApplier Transformer and @ApplyRasterRotation function
These two items allow a user to apply a pre-defined raster rotation to a raster feature.
We (Safe) use this function in various places, and have exposed it so that users may also make use of it.
The reasoning behind this is slightly complex - even to a raster user - so here’s a brief outline.
A rotated raster feature - for example a feature from Oracle GeoRaster - is typically stored horizontally with an attribute that records the rotation.
When FME reads the data it keeps it in this structure until it needs to process the feature in a way where rotation would matter - for example clipping.
Sometimes you - the user - may wish to force immediate rotation of the raster in order to inspect or process the data in some way.
For example, the bounding box of a rotated raster may be greatly different to the bounding box of a raster with rotation value.
When you apply this value you can do so in a number of interpretations. As usual these include Nearest Neighbour, Bicubic, Biliniear, Average4 and Average16
Setting a nodata value for the raster is recommended since rotation usually results in areas that are outside of the main raster values.
Note that the value of the rotation is stored as a trait of the geometry when the data is read, is not accessible as a format attribute and bears no relation to fme_rotation
To set a value for this trait you can use the Rotator transformer.
4) Random Number Generation via a Web Service
For quite some time, many forward-looking FME users have suggested the future of spatial data translation and transformation lies in accessing web services. Perhaps the generation of random numbers was not quite what they had in mind, nonetheless it is a good example of a web service that FME can access to carry out a specific (and perhaps specialist) task.
Chief FME Inquisitor Dmitri Bagh has been experimenting with this particular web service and has some great examples.
See http://www.fmepedia.com/index.php/Using_Random.org_service for an index to his workspaces and transformers.
The key benefits of this service - over the in-built RandomNumberGenerator transformer - are:
True Randomness: Dmitri explains why generators such as are found in TCL are not truly random
Random Strings: Strings include characters as well as numbers, making this great for Alpha-Numeric IDs or random passwords
Unique Values: A set of guaranteed unique values is a useful tool for assigning non-duplicate IDs (or just picking lottery numbers)
Dmitri’s transformers can generate either a number of new features with random values, or assign random values to existing features.
The really clever part of his work is to get all of the required values with a single HTTP request; the reason being that 1 request for 1,000 random numbers is way quicker that 1,000 requests for a single number (2 seconds as opposed to 7 minutes)! It’s a very interesting case of FME Best Practice.
Kudos to: The folks at random.org for making such a great service available
5) New Format: Directory and File Pathnames Reader
An experienced FME user will recognize that a new feature in one part of FME - although important in that area - can often be used for many other purposes. Often the uses to which a new function is put are not something that we at Safe had even considered. I’ve a feeling that the Directory and File Pathnames format will turn out to be such a case.
This new format takes a specified folder and returns a list of all its sub-folders, files, or both.
A feature is output for each file/folder in the list, the attributes of which include path name, file name, file extension and type (directory or file).
The intended use of this is for some component of FME Server.
However, I’ve a strong feeling that some of our more experimental users will find different ways to use this; particularly in conjunction with the RasterReader or AttributeFileReader transformer. Maybe as a QA tool, custom logger or… who knows what?!
6) FME Group on Facebook
Any FME user who is a member of Facebook may be interested to know that there is a specific Facebook group set up by some of our FME users.
It looks like a great place to communicate with users on a more personal level, for example sharing photos of Safe related travels or activities.
The link to the group is: http://www.facebook.com/group.php?gid=8670341457
7) Data Cleanup and Validation
If you have browsed the Safe blog recently, you’ll have noticed an article by Pro Services head Mark Stoakes on using FME for data cleanup and quality assurance.
Outside of format translation, this must be one of the top uses of FME, so this is a very important subject.
The article - available here as a PDF - explains to what degree FME meets all of the different requirements of the IS19100 international quality standards, and highlights some example projects where the Pro Services department has produced custom solutions for various clients and data types.
For more information on data validation solutions and other services offered by Safe’s Professional Services team, please visit Safe’s website www.safe.com/services.
8) FME Artwork Example
This egg-box design was created by Dmitri as he experimented with some 3D surfaces.
Dmitri says: “I wanted to make an example of a TriangleStrip. I made a box, tiled it, replaced with points, assigned elevation as the product of a row and a column multiplied first by sin or cos, and built TIN. Now I wonder whether I can use CSGBuilder for making eggs :) I’ll let you know.
You can find the whole set of FME inspired art work on fmepedia at:
http://www.fmepedia.com/index.php/Category:FME_Art
Brief Notes
- Long format names continued to be tweaked to find a consistent naming structure - but should all be finished now
- A fix is implemented for the Concatenator transformer to prevent attributes being wrongly published when a custom transformer is duplicated
- There have been a couple of WMS fixes to make sure SRS and HTTP Proxies are all being used correctly
- Unicode handling for the S-57 reader has been implemented
- The Oracle Raster Writer now allows easier pyramid creation by using default values for settings the user has not specified
- A double-click on a Workbench published parameter now brings up a dialog to edit the value, rather than define the parameter. Right-Click > Edit will now bring up the parameter editing dialog. This is a change in behaviour that may catch a few users by surprise.
This week’s Weekly was written to the tune (if you can call it that) of…
Florence Foster Jenkins singing the Queen of the Night aria from Mozart’s Magic Flute.
This YouTube excerpt is only 3′ 43″ long - but it will take a strong pair of ears to survive to the end.
February 4th, 2008
RSS 2.0
