Posts filed under 'CAD'
Introduction
Today’s email is an ad-hoc info session on the FME2009 beta. Expect more follow ups in the near future.
Cheers,
Mark
Contents
1) First FME2009 Beta now Released!
2) New GUI Interface
3) Transformer Changes
4) Performance Issues
5) Cool Format Updates
6) Cool Functionality Updates
7) The Complete What’s Great
1) First FME2009 Beta now Released!
Yes. You heard it here first.
The first FME2009 beta build is going to be released in the next couple of hours, if not already.
Hence the need to brief you all on the changes you will find.
As usual you can obtain it from our web site downloads page, or directly from our ftp site.
As with any beta version, be aware that it is subject to change and the functionality might not be the final version.
We don’t really recommend using a beta for production work unless there’s a specific bug fix that you need (and since most of those have been ported to an updated 2008 build, there’s even less excuse).
2) New GUI Interface
The biggest thing that you’ll notice about this version of FME is a completely overhauled user interface for settings boxes.
You may have noticed a couple of these sneak into FME2008 (for example the ‘Add Published Parameter’ dialog), but now the entire bunch of settings dialogs have been overhauled to use this design.
The Bufferer transformer is a good example. The new dialog looks like this (below):
Notice that required fields are all highlighted in yellow, to show you where to fill in compulsory values, and that you are also prompted on which type of data to enter (eg “Floating Point Value”, “String or Attribute Name”, etc).
It’s not obvious from a static image, but these dialogs are also capable of being resized, which is something you couldn’t do with the old dialogs - for example (below):
This update applies to both transformer settings and reader/writer settings dialogs too.
From our point of view, the really nice thing about this upgrade is that we now have a whole lot more options for advanced GUI features (for example greying out one setting based on the value of another) which will eventually lead to a more logical interface. Another benefit is that these dialogs are compatible with Linux operating systems, bringing us one step closer to relealizing a full FME version for Linux.
3) Transformer Changes
If you are looking for the LineGeneralizer or AreaGeneralizer transformer, you’ll find that we’ve combined them into a single transformer called the Generalizer.
Likewise, if you are looking for the Rasterizer transformer, be aware that it’s been split up into the ImageRasterizer and NumericRasterizer.
4) Performance Issues
You need to be aware that our in-house test suite has identified a number of performance issues that are still being addressed.
a) Six of our tests required a reduced value for the “Max Features in Memory” setting before they would complete. That’s six tests out of several thousand, but it’s possible that you may need to adjust this setting to get an existing workspace to complete.
b) Performance (speed) suffered when we moved to a new programming environment. We know why and it’s being worked on, but for the time being some translations may run up to 50% slower.
c) We appear to take a performance hit when reading/writing networked datasets (ie files that are not on the local filesystem). Again, we’re looking into the causes and will fix it asap.
5) Cool Format Updates
There have been a number of cool updates to formats that are worth mentioning here in brief:
a) DGN Tag Writing
Here’s something you’ll not have seen before (below). A DGN destination feature type with attributes on it.
Workspaces writing to DGN will now generate an attribute schema, and the attributes will be written to tags.
Left: Tag writing is turned on/off using a setting in the navigation pane. Default is currently on.
b) MapInfo MDB/XLS Support
The MapInfo MITAB reader now supports MapInfo TAB datasets where attributes are stored in either an MDB or XLS spreadsheet.
c) Adobe PDF Updates
PDF output now permits control over background colour, page size and ambient/specular colours.
d) Database SQL Statements
Database formats that permit the concept now support multiple before/after SQL statements. A semicolon is used for the delimiter between statements.
e) BigTIFF Support
The GeoTIFF writer now supports the ability to write BigTIFF datasets.
f) Autocad DWG MPolygon support
The DWG reader now supports MPolygon entities.
6) Cool Functionality Updates
There have also been a number of cool updates to FME functionality that are worth mentioning here in brief:
a) Arc Stroking
The stroking of arcs (well actually any curve) can now be controlled by a new directive FME_STROKE_MAX_DEVIATION which defines the maximum distance between the arc and the stroked chord. This is exposed in the Workbench GUI and in various transformers. It’s key use is to control internal operations - for example when writing to a non arc-supporting format.
b) Raster Clipping
The Raster Clipper now supports non-rectangular clipping. This is way way way cool.
c) Raster Expression Evaluator
This new transformer lets you carry out algebra type functions on raster cells.
Dmitri has a great page full of examples.
7) The Complete What’s Great
Don’t forget that the full list of Great items can be found on fmepedia:
Also check out Dmitri’s fmepedia pages for lots of examples.
http://www.fmepedia.com/index.php/Category:Dmitris_Raster_Studio
http://www.fmepedia.com/index.php/Category:Dmitris_3D_Place
http://www.fmepedia.com/index.php/Category:Web_Services
This has been a general overview of what you can expect to find in the new FME2009 beta.
The evangelism email will look at all of these new functions and formats in closer detail in the near future.
Don’t forget to email us feedback and suggestions for any of this new functionality.
June 2nd, 2008
Introduction
This “week’s” issue has important news for Linux, DGN and SQL Server users, plus a number of examples created using FME2008.
Also - this email coincides with the release of a new update to FME2008. This one is build 5204. You can get it from our ftp site at: ftp://ftp.safe.com/fme/beta/fme5204.exe
Although there aren’t any large-scale fixes - just a number of minor items that might affect a small number of users - there is a big enhancement with the ability to Write DGN Tags.
See item #2 below for more information on Tag writing, and check out fmepedia for the full list of updates: http://www.fmepedia.com/index.php/FME2008_Post-CD_Fixes
Cheers
Mark
Contents
1. The first 64-Bit FME!
2. DGN Tag writing in FME2008
3. FME and SQL Server Spatial May CTP (CTP-6 Refresh)
4. Checking for Self-Intersecting Lines
5. Weather Maps
6. VolumeCalculator
7. Neighbor Attribute Average
8. Articles on Safe and FME
1) The first 64-Bit FME!
In case you hadn’t noticed (not many people at Safe did!) the first 64-bit version of FME was released for FME2008. It is for Linux only (lucky penguin) and you can find it on the installation CD and on our FTP site.
Note, this is a true 64-bit version, not just one compatible with a 64-bit machine. The ordinary 32-bit version of FME for Linux (called linux-x86) will run on both x86 and x64 versions of Linux.
This 64-bit version will only run on Linux x64.
Also, where the 32-bit Linux FME is built for RedHat EL4 (x86), the 64-bit version is built for RedHat EL5 (x64).
Tux Picture (c) Larry Ewing
2) DGN Tag writing in FME2008
Anyone paying attention to the “What’s Great” page for FME2009 may have noticed that we have added tag writing functionality to that version of FME. For the non-CAD world, tags are a DGN format means of storing small amounts of attribute data.
On this topic there are two bits of excellent news for all you DGN users:
a) Tag writing has been back-ported into the updated version of FME2008 (build 5204). It was just so requested an item we had to do it.
This is what we’re calling ‘phase 1′ of this functionality. You need to create a list attribute for each tagset, each element in that list becoming a tag when you write the output. These would be the same tags you see when reading a dataset with tags.
There is also an option (keyword) to turn this functionality on and off.
There is an example of writing tags using FME2008 on fmepedia at:
http://www.fmepedia.com/index.php/Writing_DGN_Tags
b) For ‘phase 2′ of tag writing, FME2009 will allow you to add user attributes to your destination feature types (a new experience for you DGN users) and will (optionally) write these ‘attributes’ as tags on your DGN data.
More information on that functionality when a beta build is made available.
3) FME and SQL Server Spatial May CTP (CTP-6 Refresh)
If you read Spatial Ed (the blog page for Ed Katibah of Microsoft) you’ll already be aware that an upcoming release of SQL Server will swap the coordinate order for well-known text (WKT) and well-known binary (WKB) formats from latitude-longitude to the customer requested longitude-latitude.
So how will this affect FME users?
Well, if you upgrade SQL Server to the next CTP (for those not in the know CTP is a fancy acronym for beta version) then you should also upgrade your version of FME to FME2008 build 5200 or newer. You can read the list of updates made to that version on fmepedia and find the download on our ftp site.
Those sticking to the older CTP release of SQL Server should experience no problems, even if you still upgrade your FME. The new FME checks the version of SQL Server and acts accordingly.
4) Checking for Self-Intersecting Lines
As many of you will know, FME has a SelfIntersector transformer to remove self-intersections from lines.
Many users (and a few Safers) had always bemoaned the fact it intersects the data and won’t just check for self-intersections. The only way to do this was with a custom transformer like this one on fmepedia.
However, I’ve just found that the GeometryValidator transformer will test line features and flag as a failure any that self-intersect.
Who knew? Obviously not me.
Right: Yes, this line feature fails the “Simple” test and may therefore self-intersect!
5) Weather Maps
Dmitri Bagh - the “Q” of Safe Software - has been busy playing with web services again.
This time he’s dipped into a JSON service that returns information from weather stations within a user-defined bounding box.
It’s a useful example for anyone tasked with reading a JSON or GeoJSON dataset as it deals with the data using JSONExploder and JSONExtractor transformers. The bounding box parameters can - of course - be created using an existing dataset and the BoundsExtractor transformer, which makes this workspace flexible in terms of the area of interest.
You can find Dmitri’s workspace on fmepedia.
6) VolumeCalculator
Safe pro-services dude Dave Campanas recently came up with a useful custom transformer for calculating volume.
The volume is calculated using a set of 2.5D input features and a user-defined “base elevation”; so for calculating the difference between two surfaces you’d want to calculate their volumes separately (to the same base elevation) then simply subtract one from the other.
The transformer also accepts a boundary feature by which to define the area of interest.
http://www.fmepedia.com/index.php/VolumeCalculator
7) Neighbor Attribute Average
As for me, here’s my contribution to the fmepedia workspace pool: a Neighbor Attribute Average Calculator.
This workspace takes a set of adjoining polygons, examines a user-specified attribute and for each polygon calculates the average value of its neighbors.
One use - and I think this is what the requestor wanted it for - would be to identify spikes and wells in a DEM dataset. If the Z value of a polygon is greatly more or less than its neighbors average then it must be an erroneous value.
Do you have any other uses? Let us know.
http://www.fmepedia.com/index.php/Neighbor_Attribute_Average
8) Articles on Safe and FME
Check out the latest edition of GeoInformatics (p24) for an article by Safe president Don Murray on ‘The Growing Need for Spatial ETL’.
http://fluidbook.microdesign.nl/geoinformatics/03-2008/
Does the cover image look familiar?
If that weren’t enough, Dale also got his writer’s hat on and has two blog postings.
The first is about KML becoming an official OGC specification, and how its impact is comparable to the shockwaves felt when Wayne Gretzky went to play hockey for Los Angeles.
http://spatial-etl.blogspot.com/2008/05/kml-is-ogc-open-specification.html
The second is on the recent Where 2.0 show in San Francisco.
It tells how even if you know nano-formats, you might still not know Jack.
http://spatial-etl.blogspot.com/2008/05/safe-on-location-at-where-20.html
Upcoming Events
Star-Apic (an FME reseller) are holding a “Managing and Publishing Spatial Data” event at the Ordnance Survey offices in Southampton, UK.
Included in the events are a workshop on “Making the Most of FME”. See the Star-Apic web site for more details.
UK reseller Dotted Eyes will be running a FME event at their offices in Bromsgrove, Worcestershire on 18th June 2008. Email marketing@dottedeyes.com for more information.
Now that Celine Dion has finished her Las Vegas show, fellow-Canadian Dale Lutz is taking over with a three day stint at the Intergraph 2008 Conference. June 2nd-5th are the dates. Booth #805 and a 45 minute talk on Spatial ETL for GeoMedia.
For more upcoming events with Safe Software participation, see our web site: http://www.safe.com/aboutus/events/tradeshows.php
Coming Soon
The FME Evangelist - this weekly in a more friendly (and email deliverable) form
FME User Central - a new access point for all FME technical resources…
This week’s Weekly was written to the tune of…
…lots of things, but mostly the Guardian Football Weekly podcast.
In the latest issue the pod discuss John Terry, Avram Grant’s replacement and the play-off finals.
If football (soccer) isn’t your style then try the BBC’s Friday Night Comedy podcast instead.
May 26th, 2008
Contents
- FME 2008 Release Updates
- AutoCAD Map 3D Object Data Reading Examples
- Shape Datasets and the Age of Innocence
- Google Spreadsheets Reader/Writer
- Creating a Shape Index with FME
- OIDs in PostGIS
- Measurement Unit Converters
1) FME 2008 Release Updates
The first of the (previously mentioned) FME2008 Update builds is now available for download. Remember, these are minor fixes made available in lieu of an FME2009 beta.
You can download the first FME2008 update online (it’s now the default 2008 download) at:
http://www.safe.com/support/downloads.php
You can find a list of the changes in each particular build (this one is build 5200) on fmepedia at:
http://www.fmepedia.com/index.php/FME2008_Post-CD_Fixes
I’d suggest you read the list of updates before automatically installing this build, since only a very small minority of our users would be likely to find any benefit in the new build; but if you do have any questions don’t hesitate to contact support@safe.com
2) AutoCAD Map3D Object Data Reading - Example Workspaces
AutoCAD Map3D Object Data is a very flexible and open-ended format, which is great for you users but a challenge for us when creating the reader.
What we did was add one fairly unusual setting - a reading mode (right) - that affects how the source schema is displayed in your workspace.
Is essence you can get different views of the data model depending on what actions you want to carry out on the data.
To explain the different modes we created a set of example workspaces, based on the same dataset but each in a different reading mode.
These demonstrate the different results you can obtain with each mode and help explain when you would want to use them.
See the examples on fmepedia at: http://www.fmepedia.com/index.php/AutoCAD_Map_3D_Object_Data_Reading
3) Shape Datasets and the Age of Innocence
The Safe support team has just received their first Shape dataset more than 2GB in size. As Dale puts it, the age of innocence is over. The content below explains why.
NB: If your eyes glaze over at technical details then skip the bits marked with ###.
What is the Problem?
In theory a Shape dataset greater than 2GB in size is - if not impossible - certainly very implausible and difficult to handle. It should be noted that when I refer to a Shape dataset I mean the .shp part (the .dbf part has it’s own 2GB limit which is another problem by itself…)
### In technical terms this is because internal pointers between the index (shx) and spatial data (shp) are usually stored as signed 32-bit integers, so any size over 2^31 (2^31=2147483648 or 2GB) would need a larger number than that can store. So a computer application writing Shape data would find its counter overflows at (2^31)+1 and - if it didn’t crash at that point - start writing invalid pointers. ###
However, any software that writes a Shape file might ignore the problem, and just churn out bad indexes, which is how invalid datasets more than 2GB in size come to exist.
What does FME do?
When FME reaches the 2GB maximum it too ignores the problem (we were obviously still innocent when we implemented it) and continues to write data. However - while that leads to bad index pointers, they are bad in a way that is predicatable. So we made a change to our reader to handle this and now (in FME2008 build 5199 or greater) FME is able to read back any oversize datasets created by FME. We’d also be able to read Shape datasets from other naive applications that wrote data in the same way.
That sounds like a workaround. Is there anything else you can do?
Good question. ### Getting technical again, Shape internal pointers are actually measured in “words”. A “word” is a unit of data that depends on a computer’s architecture. For backwards compatibility reasons a 32-bit computer usually uses 16-bit words. So FME counts using a 32-bit signed integer but then divides by two to get the correct pointer. If you’re following along you might now say, “Ah! Since you divide everything by two, there’s no reaon why you can’t double the size of the pointers”; and you’d be right. We just can’t do the counting using a 32-bit integer, as an application usually would, because that would overflow at the 2GB mark. ###
So a future version of FME (FME2009, build 5525+) will be able to write up to 4GB Shape (again that’s .shp not .dbf) files. A fixed limit of 4GB will be hard-coded so we don’t get datasets any larger, which really would be invalid.
Not only will FME be able to read these 4GB datasets back, we believe the data is the format most likely to be readable by other applications.
Any problems to look out for?
Yes! Technically our theory is sound - and not contrary to the official Shape specification - just complex and relatively obscure. I don’t know of another application that is using this technique so we can’t guarantee that these applications (i.e. those still sweet and innocent) would be able to read our oversized datasets correctly, nor can we guarantee that we can read an oversized dataset created by another application (though even if they’ve created invalid data, if it’s in a predictable way, there’s a good chance we will).
Will a 64-bit computer support 4GB+ files?
Yes, but it would need a change in the Shape specification, along the same lines as recent changes in the TIFF format.
The TIFF format also had a size limit [### 4GB, so presumably they were using unsigned 32-bit pointers as opposed to signed with the 2GB-limited Shape ###] and so a new specification - BigTIFF - was created to allow Large File Support (LFS). Similar updates would need to occur to the Shape format to permit files greater than 4GB in size.
### So with (unsigned) 64-bit pointers we get 2^64 = 17 million TB - that’s Terabytes not Gigabytes folks, and even the most avid data collector is unlikely to have a Shape dataset that size. To put it in perspective, this is equivalent to 16,384 Petabytes (apparently Microsoft’s Virtual Earth imagery dataset is 14 petabytes in size) or 16 exabytes. According to wikipedia 16 exabytes of disk space would cost you $3 billion at today’s prices! ###
For the techno-confused amongst you (that includes me) here are some useful explanatory articles on wikipedia:
http://en.wikipedia.org/wiki/Large_file_support
http://en.wikipedia.org/wiki/Gigabyte
http://en.wikipedia.org/wiki/Word_%28computing%29
Now if you ever appear on Jeopardy you can feel confident in saying, “I’d like Shape datasets for one thousand please Alex”.
4) Google Spreadsheets Reader/Writer
If you weren’t aware, Google makes a Python API available for reading and writing Google data - ie data on any of the Google web services such as Documents, Picasa, YouTube (I didn’t even know that was a Google service), Calendar, etc
Pro-services dude Aaron has taken advantage of this - and FME’s ability to run Python scripts - to create an FME reader/writer (actually custom transformers) for Google Spreadsheets.
There are two versions of the Writer; full and fast.
The ‘full’ version writes features one at a time. It is slow but enables the use of multiple worksheets in a single spreadsheet.
The ‘fast’ version writes to a CSV file and uploads it as a new spreadsheet (but cannot write to an existing spreadsheet).
This is another great example of using FME to access web services.
For more details get the download from fmepedia at:
http://www.fmepedia.com/index.php/Google_Spreadsheets_Python_Read-Writelet
5) Creating a Shape Index with FME
A frequent question to the FME support team is, “How can I create a spatial index when I write a Shape dataset?” That’s something that previously wasn’t thought possible with FME, but now a combination of Python script and ArcObjects will let you do this.
How important would it be to have indexing when your Shape file is 4GB in size, eh?!
The key is that an ArcObjects Python script can index Shape files, and FME can run Python scripts passing into it such useful values as ‘destination dataset’. So by adding such a script as a Shutdown Python Script, it’s possible to create a spatial index for any dataset you have just written.
You can find an example workspace demonstrating this technique on fmepedia at:
http://www.fmepedia.com/index.php/Spatial_Indexes_for_ESRI_Shape_datasets
It’s simpler than it sounds because you just need to copy the Python script into the Shutdown script setting dialog (Navigator Pane > Workspace Settings > Advanced) and away you go. You can use the script in our workspace, but what’s really nice is that you can export a Python script from almost any tool in ArcToolbox and use that to do any number of things!
A few quick notes:
- You’ll need Python installed to run this (unlike TCL it isn’t included with FME). It can be found here.
- You’ll need ArcObjects installed (or at least the ArcGIS scripting library), which means you need an ESRI product such as ArcGIS.
- We didn’t get great results with Python v2.5 and ArcGIS v9.2, so we’d suggest using Python v2.4.
- A minor FME/Python fault means you can’t run a shutdown script without a startup script (at least not one that accesses parameters). So make sure to create a startup Python script in your workspace, even if it is empty.
No promises, but this functionality may even find its way into FME as a properly supported parameter! Remember, you heard it here first.
6) OIDs in PostGIS
Here’s a question and answer on the FMETalk user group that is probably worth passing on.
On 01/04/2008, Jeff wrote:
> Hi there,
> Could someone point out how to get FME to see the postgis oid column
> (created by FME when you choose ‘with oids’ in the format parameters)
> in the source features?
On 02/04/2008, Roland wrote:
> Morning Jeff,
> You can put in your own select statement in the feature properties.
> Under the ‘Parameters’ tab you’ll find inputs for WHERE clause and SELECT statement.
> In the SELECT box type in something like: select oid,* from <table_name>
> Attach an AttributeExposer with [an attribute] called ‘oid’ so that you can see it, and you’ll have OIDs (sounds painful).
Thanks to Jeff for posting the question and Roland for answering it; plus thanks to everyone else on the user group who participates in the FME community in this way.
7) Measurement Unit Converters
Safer Aaron Koning has really been busy (at salary review time - coincidence?!) - he’s also produced a series of custom transformers that translates length, area or volume attributes from one set of units into another.
This is a nice example of lots of things - using a Custom (Independent) published parameter, creating a custom transformer, use of the ValueMapper transformer, etc. I think I will add these to the FMEData sample dataset.
It’s not just yards or metres that it will convert either; there are lots of different units such as miles, chains, hectares, acres and six different types of foot measurement!
So if you ever wanted to convert your LengthCalculator results from your current units into twips, light-years or angstroms, here’s your chance!
Check out fmepedia to get these great custom transformers:
http://fmepedia.com/index.php/Measurement_Unit_Converters
Brief Notes
- The Spring 2008 edition of the FME Insider newsletter is now available. Apologies to the southern hemisphere for whom it is anything but Spring!
- An Austrian FME user conference will take place in Vienna on 20th May 2008. Click here for more details.
- Interested in Mastermap and/or SQL Server Spatial? See this blog posting for a great example that uses both.
- A month or two old, but here’s a blog posting about Don Murray’s visit to a conference on Geospatial Data Standards.
This week’s Weekly was written to the tune of…
Gerry Rafferty’s superb album North and South.
It’s my favourite of his albums, but relatively unknown. The only track I can find on YouTube is “Nothin’ Ever Happens Down Here“.
May 5th, 2008
Contents
1) GeoJSON Support
A very quick heads up that the latest FME betas include support for GeoJSON v5. Also the .json extension is now recognized by FME as a supported dataset. For more details, check out http://wiki.geojson.org/GeoJSON_draft_version_5.
2) Object Data Writing
There’s been a lot of noise recently about AutoCAD Object Data and FME, and you may be wondering, “why is this so significant?”
Well, as I understand it, Object Data is a way of storing attribute information inside a DWG file, and is therefore Autodesk’s way of providing intelligence to their native format. This increases its ability as a format for uses other then CAD (as well as increasing the options to which a CAD system can use it).
If you type autocad object data into Google the first line that comes up is… “Object Data can only be created using Autodesk Map, Land Desktop or Civil 3D”.
So being able to read and write this type of data outside of an Autodesk product is a very significant fact indeed!
Safe has been working very closely with Autodesk to enable this functionality and is justifiably proud of how it has implemented this technology within the existing FME framework.
Anyway, as of build 5149 a beta version of this writer has been exposed in FME, as Safe’s Christmas gift to its AutoCAD users! Enjoy.
3) Raster Updates - New Transformers and Interpretation
This week sees a number of new items of raster functionality.
Although some originally vector transformers are capable of handling raster, some new transformers - being dedicated to raster - are more user friendly in that regard.
RasterTiler
The new RasterTiler transformer divides a raster feature up into a number of smaller ’tiles’.
A user would do this to make raster data management and viewing easier, by working with many smaller raster images and not one gigantic image.
From a user perspective, the big difference between this and the standard Tiler transformer (which can also accept raster) is that you get to choose how many rows and columns of tiles you want, and FME does all the work. To get 4×4 tiles using the regular Tiler you would need to know the size of the raster in pixels, divide by 4 and enter that number.
An example can be found on fmepedia at:
http://www.fmepedia.com/index.php/RasterTiler_Example
RasterPyramider
The new RasterPyramider transformer creates pyramids of raster data; ie layers of rasters with different resolutions.
When viewing most types of spatial data a key concept is that you view the coarse data when zoomed out, but finer data when you zoom in.
If there was such a book as Raster Pyramiding for Dummies it would say “This technique is similar to the one used by Google Earth - open Google Earth and the data is very coarse, because to show the highest quality data when viewing the entire Earth would cause major performance issues, but zoom in and it gets refined because you are looking at less surface area. The slight delay in display when zooming in (and the status bar says “Streaming”) is Google loading larger pyramid levels that have a better image quality and resolution.”
So what this transformer does is turn a raster feature into a set of layers of varying resolution that you can use in a raster viewer to improve performance.
An example workspace using the RasterPyramider can be found on fmepedia at:
http://www.fmepedia.com/index.php/RasterPyramiderxample
When you have just one raster feature that you want to both pyramid and tile then you would use the Pyramider first then the Tiler to divide the data up (the number of tiles would relate to the layer number).
An example workspace using both Pyramider and Tiler can be found on fmepedia at:
http://www.fmepedia.com/index.php/RasterPyramiding_Example
RasterSubsetter
The RasterSubsetter allows a user to create a subset of a raster - in effect extract, or clip out the centre area - defined in pixels (or cells). This is different to the existing clip and tile functions which only operate on a spatial reference system.
An example can be found on fmepedia at:
http://www.fmepedia.com/index.php/RasterSubsetter_Example
Raster Interpretation Methods
Previously FME had three interpretation methods - Nearest Neighbor, Bilinear and Bicubic.
Now it also has Average 4 and Average 16
Average4 and Average16 are types of averaging filter useful on numeric rasters such as DEMs.
Average4 averages each pixel according to the values of the 4 adjacent pixels, and Average16 presumably does the same but on a wider spread of data.
The idea is to create a smoother surface and reduce spikes or sharp depressions. If you tried this on a color (image) raster they would have the effect of slightly blurring the data.
These methods are applicable in any transformer using some form of interpretation; for example reprojection, resampling, pyramiding, etc
4) FMETalk User Group
I just wanted to quickly highlight the fact that the FME User group has had its busiest year… ever!
By my count the current number of messages for 2008 is 2250 (and rising) beating the previous best (2002 - 2181) by some way.
Last month (November) was also the busiest month ever… 260 messages.
Thanks to all users and Safe staff who contribute to this useful tool.
If you (or your FME customers) aren’t a member - join now at:
http://groups.google.com/group/fmetalk
5) Tester and ExpressionEvaluator in Custom Transformers
Harbinder in our development team tells me that automatic publishing of parameters (when you first duplicate a custom transformer) will now check the attributes used in Testers and ExpressionEvaluators to see if the names are already exposed - if so the transformer parameters will not be published.
This improvement (build 5153+) should please users who are big on custom transformers because it makes the runtime dialog a lot easier to understand.
6) See What We Can Do With Solids
As an example of using FME solids, see this Dmitri-generated screenshot of a world DEM, as depicted in ArcScene.
Read on for some Safe holiday poetry… FME Poetry
Since it’s a holiday season I hope you’ll indulge me evangelising in prose this week, with a poem inspired by the Dr Seuss classic Green Eggs and Ham…
After reading this, even your children will understand FME (and want one for Christmas).
Do you use our
F-M-E?
I do not use your
F-M-E.
Your data could be here or there,
Your data could be anywhere!
My data’s here, on this CD,
so that does not apply to me,
I do not need your F-M-E!
Your data could be from Japan,
Spain, China, France, or Vietnam!
My data’s only English see,
What’s more it’s all in plain ASCII,
And it’s all here on this CD,
So I’ve no use for F-M-E.
Your data might be polylines,
paths or rasters, text or splines!
My data’s single points, 2D,
And like I said, in plain ASCII,
And it’s on one English CD,
So I shan’t want your F-M-E!
Your data, it might need transforming,
Cleaning up, to be conforming.
My data’s simple A-B-C,
And that’s the way it’s meant to be,
Unchanging points that aren’t 3D,
So where’s the role for F-M-E?
Your data you might need to fix,
To join it up to Informix,
My data’s all one table, see?
It doesn’t have a foreign key,
An outer join’s no use to me,
So I won’t need your F-M-E!
Your data could be MID and MIF,
Shape, Idrisi, GeoTIFF!
Oracle or Map 3D,
S-D-F or C-S-V,
N-T-X or Smallworld 4,
Shape, or SAIF or hundreds more!
Our data’s not proprietry,
We use a shareware app for free,
Inter-what-ability?
What’s the point in F-M-E?
But what about the need to train?
Don’t your staff ever complain?
[Ye Gods - this man has half a brain]
My staff all have a PhD,
And they can’t train, they’re too busy,
In fixing data quality,
There’s no time for F-M-E!
What if your data….. oh, but hark!
Is that the trilling of a lark?
No it’s my Blackberry…
A call from our most valued customer…
…you want what?
…when?
But…
That data could be here or there,
That data could be anywhere!
That data could be from Japan,
Spain, China, France, or Vietnam!
That data might be polylines,
paths or rasters, text or splines!
That data, it might need
transforming,
Cleaning up, to be conforming.
That data we might need to fix,
To join it up to Informix,
That data could be MID and MIF,
Shape, Idrisi, GeoTIFF!
Oracle or Map 3D,
S-D-F or C-S-V,
N-T-X or Smallworld 4,
Shape, or SAIF or hundreds more!
And what about the need to train?
My staff are going to complain!
[Ye Gods - I must have half a brain]
mmmmmm…
OK. I will try your F-M-E.
Can I be an evaluee?
Wow!
My F-M-E I’ll never trade,
Now I’ve drunk that cool kool-aid
I’ll use it when my data’s there,
Or when my data’s anywhere
I’ll read that data from Japan,
And Unicode from Vietnam
And if my data’s awkward splines,
Workbench makes me polylines
And when my data won’t conform,
Transformers turn it to the norm
And if my Informix is full,
We’ll join instead to Oracle
Translating some S-57,
Is truly a foretaste of heaven
But if my raster’s SID or SPOT,
F-M-E won’t care a jot
And it’s so easy for to learn,
We’ll finish quick, with time to burn
YES! I love your F-M-E,
Thank you
Thank you
I’m filled with glee!
December 21st, 2007
1) @Tcl2 Temporary Filenames
This is a new ability (build 5147+) with our @Tcl2 function to generate temporary file names within a TCL script. The syntax is FME_TempFilename [] []
While this will be useful to some users, for Safe Software, it allowed development of a new transformer….. the ImageFetcher…
2) ImageFetcher Transformer
This new transformer (build 5147+) - when handed the URL of a TIFF, PNG, GIF or JPEG image - will return that image as a raster feature. The raster feature can then, of course, be further processed or converted in any FME supported manner.
If an error occurs and ‘Continue On Error’ is set to Yes, the output feature will have no geometry and the error message will be logged, but the translation will continue. However, if an error occurs and ‘Continue On Error’ is set to No, the translation will fail.
Above: An image on the Safe website - and as read, written to GeoTIFF and opened in the FME Viewer.
This example can be found on fmepedia at: http://www.fmepedia.com/index.php/ImageFetcher_Example
Again, this will be very useful in its own right, but is really special when combined with another new transformer… the WebCharter…
3) WebCharter Transformer
Just last weekend Google released a new web service through which you can submit numeric data to create a chart - much like you can do with an Excel spreadsheet. So this new transformer (build 5147+) takes a set of data in FME, and turns it into a URL which can be submitted to this Google Chart API (http://code.google.com/apis/chart/)
One URL is created for each feature that enters the transformer. The data for the chart is taken from the list specified, and each element of the list must be numeric. Only one data series is currently supported (i.e. there is no ability to overlay two numeric sets of data on the same chart).
The transformer supports a subset of the Google chart types: Line charts, Bar Charts, and Pie Charts.
How do you submit that URL? With the new ImageFetcher of course!
But also, any format that includes the ability to embed a URL - such as a link on a KML placemark - in effect lets you add a dynamic chart to the data!
Left: Interopolis parks data, converted to a Google Chart URL, read with the ImageFetcher, written to GeoTIFF and opened in the FME Viewer.
This example can be found on fmepedia at: http://www.fmepedia.com/index.php/WebCharter_Example
4) HTTPS Support
On the web theme, a number of users have been asking if they can read data from HTTPS URLs - as opposed to an open HTTP site. The answer is yes …. but for the moment Safe has to be very careful about how and where we ship it.
So if you have a burning need for this, then please email support@safe.com with your request and we’ll work something out.
5) FME on YouTube
I’m sure you all were wondering why we didn’t have it before, so here you are - a YouTube channel for FME users. Since there is a choice of account type - one of which is ‘Guru’ - the group is named the FME Guru.
At the moment, we only have one movie up there - the transformer drag-and-insert demo - but we hope to get some of Ryan Cragg’s excellent movies there soon, the video of GIS day, any small demo movies we do in the future, and perhaps some training exercises (time permitting).
See: http://youtube.com/FMEGuru
6) New GeoConcept Support
The current FME2008 beta version has an updated reader to read GeoConcept data. For those who aren’t familiar with this format, GeoConcept is a French GIS application
The reader updates are based on new functionality provided by GeoConcept. The key benefits are that the reader is a lot faster (3x faster in recent tests) and that you no longer need to have GeoConcept installed to make use of the reader.
The new reader also supports GeoConcept v6, but is (with a few small differences) fully backwards compatible with older versions.
7) AutoCAD_OD Writing
If you thought FME’s AutoCAD Object Data reading was impressive, then wait till you see writing! In a sneak preview of this functionality I was able to see how we can convert attribute data into object data - so easily you wonder what all the fuss is about!!!
Breaking News: In the background I can hear Dr. Lutzenstein saying, “Igor! Throw the switch Igor” which means this should be in a beta by this time next week.
8) Advanced Format Handling
There are a couple of components for handling format reading and writing at an advanced level which are worth mentioning. Both of these emphasize how workspaces might be generated by a developer but used by a lower level user without them having to be aware of the technology used.
The generic writer is a destination writer whose format is not determined until the translation is actually carried out. This allows a developer to set up a generic workspace capable of writing to any format the user decides.
The “Feature Types to Read” parameter allows a user to decide which source feature types will be read during the translation. The benefit is that the user doesn’t have to edit the workspace and disable non-required source layers, they can be defined in the run-time dialog.
Although these items are not particularly new - they were in FME 2007 - they are particularly topical because they come in so useful when deployed using FME Server. And what is new is that they’ve been added to the FME training course in a special section on format translations. So anyone interested in advanced format handling with either of these functions can find screenshots and examples by taking FME 2008 training next year.
9) 3D Elements in the Creator
As of build 5145 the Creator transformer now includes the ability to create 3D related objects such as Boxes, Faces, Triangles etc.
Above: A Creator creating a box feature… [actually the coords in the screenshot look wrong... perhaps a bug where they get reset on reopening the dialog?]
Below: …the same box feature as written to PDF and opened in Adobe reader.
This week’s Evangelism Weekly was written to the tune of…
The Stylistic’s “Can’t Give You Anything”. A great, upbeat, funky Philadelphia soul track.
It’s in the Itunes store, but if you’re too cheap to pay 99c, here’s a YouTube link…
http://www.youtube.com/watch?v=LPTbzPHkEGA
Have feedback?
Email FME Evangelist, Mark Ireland at mark.ireland@safe.com.
December 14th, 2007
Next Posts
Previous Posts
RSS 2.0