April 27th, 2012
This post is a live-blog record of the FME World Tour 2012 event in Vancouver, which occurred on April 27th 2012. Presenters from Safe are Don Murray, Dale Lutz, Stewart Harper, Mita Hajirakar, and Mark Stoakes. Customer presentations are by the RCMP and Fortis (a utilities company).
To view in chronological order, scroll to the bottom of the page and work your way back up!
3:05pm: Phew! We’re all done. Thank you to anyone who happened to follow along on this blog live. Bye for now.
3:00pm: Now the lizard has found his name - but it is in Mayan script. Bah! He’ll need to watch this Nova documentary to help decipher them!
2:56pm: After finding the shortest route (ShortestPathFinder) the lizard can now set off on his quest. But first he’ll build a 3D model of the area using the DEM, some Lidar info, satellite imagery, and a vector boundary.
2:52pm: Now he uses the NeighborFinder to find info about his points of interest (GPS’d) relative to the road. Note how the NeighborFinder returns the distance to the nearest neighbour, and the attributes from the neighbour too.
2:48pm: Now the lizard needs to create a route, but first needs to validate and mend the road network data. He uses the NetworkTopologyCalculator to ensure everything connects as it should. Linear Referencing - as Mark says - is something we can do really well, but it’s not very well known. We’ll be emphasising this capability shortly.
2:45pm: The lizard now has some GPS tracks in XML format and wishes to merge them into the Geodatabase. He’s going to use the XML reader and extract information using the flatten options.
2:41pm: Next up: The lizard shares his photos using Don’s notification server demo. Then he’ll do some trip planning by mashing together some imagery, a DEM, some SketchUp datasets, and a road network.
2:37pm: The second task is to load reconnaissance photos into a Geodatabase. The technique is quite impressive (you’ll know it if you’ve done the Esri course) and I shall have to write about it someday.
2:35pm: The first task for the lizard is to plan a great circle route to his destination. He does this with a transformer downloaded from the FME Store:
2:34pm: Mark Stoakes (head of Pro Services) is up now with an Interactive Quest showing FME tips and tricks. I believe this is the last talk of the day. Mark is talking about Mayan “technology” because it is part of the quest. The quest is going to be for our lizard - the one that appears on our splashscreens - to track down his heritage and find out his true name! He is asking questions along the way, and handing out Safe Software sporks for those who know the answers.
2:29pm: Email firstname.lastname@example.org (it goes straight to Don) if you have a challenging XML dataset you need help reading or writing.
2:28pm: The key thing is that the solution handles multiple objects. If there are multiple ladies that swallow flies, it’s necessary to identify which fly is being swallowed by which lady, else the output will be nested incorrectly. So the XMLTemplater is creating about nine nested tables in order to generate the properly nested and indented XML.
2:25pm: “The old lady who swallowed a fly” is the end point of Don’s demo. It’s all about creating nested XML, which is always difficult to do when converting from “flat” data. In the workspace, the Creator transformers are just generating the data. One XMLTemplater transformer is handling the work:
2:18pm: As Don says, there is a clear difference between GML and XML that just happens to contain GML geometry. btw KML is one format that is the latter. In FME2012 we can now pull GML geometry out of an XML dataset by using the XML reader (without flattening the data), then applying the GeometryReplacer transformer. The contents the GeometryReplacer works on are stored in an attribute called xml_fragment, which can then be flattened with the XMLFlattener transformer to then extract attributes.
2:15pm: The FME updates for 2012 (for XML) are quite immense. Key ones are XMLUpdater, XMLSampleGenerator, XMLFlattener transformers. I’ve written an article about these. Hoped to post it yesterday, but it won’t be up until next week now.
2:09pm: Don is now starting his XML “Sing Along” talk. He says “no FME release is complete without making XML easier”. That doesn’t include learning XSLT, XQuery (I have the book on my desk right now) and Safe’s own xfMaps. Users shouldn’t need to have to learn that stuff.
2:05pm: The takeaway is that they have standardized their file format, have saved $$$$$$, and have received excellent technical support from Safe Software. In future they hope to automate the process in some way, and provide a self-serve option for planners.
1:49pm: Basically this was an export from Smallworld to Shape, with some transformation to create a network model, then a bunch of QA/QC. The Shape data could then be loaded into the target design system. Smallworld joins are always an issue. The data is extracted in a raw form, then merged back together with the FeatureMerger.
1:45pm: Their problems were using two different packages, one of which needed upgrading with new hardware. Migrating data to one system, when it was traditionally moved and built by hand, was going to be difficult; so they turned to FME.
1:44pm: These guys do hydraulic modelling and long range capital planning. Sort of like Sim City Harry says. Nice!
1:42pm: Next presenters up are Ted Norcott and Harry Bhandal, from Fortis, a utilities company.
1:39pm: Big News! FMETalk is a Google Group that has been online for many years now. However, shortly we’re moving this community to our own systems. Users will get points and badges for question responses. Keep watching for more info!
1:37pm: Stewart now mentions some of the changes in the way you contact support. Basically we are now suggesting folk get in touch through www.safe.com/supportrequest instead of the old email address.
The reason is that we’ve implemented keyword-matching on the request form, so that FMEpedia retrieves articles related to the words in your support request. This way we hope to solve some of your problems before you even get as far as a support case. We’ve actually got a good deflection rate already - i.e. many users find a solution without having to wait for a support response.
1:32pm: Stewart mentions this blog (thanks!) and also the FMEChannel on YouTube.
1:29pm: Stewart is going to do a talk on FMEpedia. In particular the support resources available to help users. We’re trying hard to unify all the FME resources such as FMEpedia, this blog, the user group, etc.
1:20pm: I discussed Workspace Templates before in this post. , and Robyn and I discussed Geodatabase templates in this video evangelist. Mita is now working on my specialist subject: election results.
1:18pm: The ArcGIS for Local Government model is a good example of a template Geodatabase. I guess Mita is going to create an FME Template that references the Geodatabase template.
1:16pm: Mita is up now, with some Esri Tips and Tricks. This presentation was the most popular overall on the World Tour. She is talking about Geodatabase templates and FME Workspace templates. Be sure not to get the two confused. What’s more confusing is that Esri call their templates “workspace” documents. Yikes!
1:11pm: More seriously, parallel processing is a great tool that splits transformation into multiple system processes. That way you get to maximize processor use in a multi-core computer. Again, it’s new for FME2012.
1:07pm: I will claim responsibility for the atrocious puns in Stewart’s powerpoint. Did you see the picture of an apple core and the needle being threaded? The worst is the “multiple levels of parallelism”. Get it? Sorry.
1:05pm: Stewart calls it “multiprocessing”. I called it “parallel processing”. I wonder if there is a technical difference, or are they two names for the same thing?
1:04pm: Apparently the record translation length is 4 weeks. What’s your longest translation? Let us know.
1:00pm: Welcome back. The afternoon session seems to be starting with a lightning talk on Multiprocessing in FME. Again, it’s something I’ve blogged about before.
12:00pm: It’s time for lunch. The event - and this live blog - will restart in an hour’s time.
11:56am: Don is keen to stress that none of this takes away from older uses of FME Server - this is just a set of new ways of leveraging FME Server engines. You can contact Safe for more info.
11:50am: A final demo is an email-based spatial data QA system. Don emails a CAD dataset to an FME Server notification server, which runs a QA workspace and sends a reponse about whether the data passes a set of tests or not. FME will send back either a report or the same CAD file containing flags showing the problems. If the data is good, it will be loaded into a Spatial database (Oracle in this case). Oracle also has a trigger set to alert FME Server with another notification, to push the updated data out to whatever subscribers need it:
11:47am: Maybe we could create an FME geek finding app? Every time you get within 100m of another FME’er (who is running the alerter app) you receive a response as a subscriber. Then you can get together and talk about spatial-based, real-time, FME subscription services ;-)
11:41am: Don and Dale talk about two new apps for the iPhone: FME Reporter and FME Alerter. These are example apps that show the potential of integrating location-based services with FME Server. By Dale entering an event in the app, Don gets an email to alert him
11:37am: Since FME Server now supports incoming email, you can send an email with a photo attached. FME supports raster and so, if the photo is geotagged, it can be located and timestamped and then used as an event. email@example.com is the address. Don suggests you could photo potholes for local government - rather than a site he saw that asked for intersection name, description of the hole, etc.
11:34am: And Don - as a subscriber - gets an email to alert him to it. Presumably the event is inside his defined area of interest:
11:33am: Now an event occurs, and includes a location:
11:32am: Don signs up to a subscription service in the demo, and defines a geographic area. He will receive alerts (by email) when an event occurs. Because we are using FME, that event can be spatially related to the defined area.
11:27am: Don is showing a demo. The idea is to set up a notification tool using minimal work. It’s not anything that you might not have seen before, but you can do it very, very quickly. His demo took just 2 hours to set up the infrastructure and about 10 hours to build the workspaces. This is what I’ve always said. FME Server is a lightning fast development environment. You could create spatially related project in a fraction of the time you might expect.
11:23am: Basically there are two types of “users” in this architecture. There are subscribers who want to receive data and reporters who are reporting events. These users can be human - or may be anything like a computer system, or sensor, etc. I think you can see which is which in this diagram:
11:20am: FME2012 has an Event Driven Architecture (EDA) that lets you both pull and push data. This includes sending and receiving transformation notifications. FME is so good because it ties in spatial data transformation to the whole EDA idea.
11:13am: Dale says there is more processing power in a smart phone than was used to make the first Star Wars movie. A bit of trivia I will have to take his word for.
11:11am: Don talks about Server. Remember that Server is about distributing data, and processing data in bulk. But this can be carried out in response to an event, and he is going to show how.
11:10am: Don is about to give his Power of Events presentation.
11:09am: Early voting shows InlineQuerier and FME Store transformers are leading the poll for the Lightning Talk to give. Connect to the live stream and there is a button to open up the polls.
11:05am: In questions it is revealed that Mark Stoakes (of Safe) is responsible for recommending the TCLCaller! The RCMP use both Python and TCL. The standard time zone is Pacific Standard time, although some areas are in Mountain Time. FME can help convert if necessary.
11:04am: Overall, FME has helped simplify the RCMP’s data processing. It can be made into a single script that produces the required data, and helps with variable fields. It also works well with the larger GPS files. Most importantly, it allows a fast turnaround on mapping requests. For more info you can contact firstname.lastname@example.org or email@example.com
11:03am: By the way, comments are open at the foot of the page, if you have something to say about the event or a presentation.
10:59am: The prior method was using spreadsheets! It would take 1/2 day to process a set of logs. Now FME lets them do it in 1/2 hour. A month of logs can be processed on a server (FME Server?) in 11 hours. Their workspace is nicely annotated and with bookmarks. Kudos for the best practice design. One transformer looks to have dozens of output ports, which makes me wonder if that one could be improved upon.
10:56am: Police cars are equipped with GPS receivers. They produce raw logs that can be translated and transformed with FME to help produce mapping. The logs for one day can exceed 400,000 records. Here’s what a log looks like:
10:50am: I’m asked if the event is streaming video live. Yes. See it at: www.safe.com/livestream. You will need Silverlight to view it.
10:49am: The RCMP mapping unit are responsible for spatial data analytics for BC. This includes crime maps, but also address validation, basemap maintenance, and mapping of GPS logs. I know from experience that addresses are vital to the police dispatch processes, having worked on a large project once.
10:47am: According to the powerpoint slide, the next talk is from Robert Schultz and Heidi Lee of the RCMP. I’ve no idea what they are using FME for. This will be interesting.
10:35am: To summarize. FME is about moving data. We want folk to be able to make use of it, not spend their time arguing about out of spec Shape files! There’s now a 15 minute break, both at the event and in my live-blogging.
10:28am: FME Server Event Processing. FME Server now has an email server built in. It can send and receive emails in response to jobs or events - such as from sensors…. but more from Don on that later. A demo seems to copy that from 9:37am, but is done on Server and the proof is your own email address is stamped onto the output data! Simply send an email to: firstname.lastname@example.org and you will get something like this in return:
10:21am: FME Server for 2012 now includes Active Directory integration, to improve the ease of setting up security. You can basically use the same security as on your local systems. You can create groups and assign users to them as well.
10:17am: Moving onto Server. A poll question shows most folk know about Server, even if they aren’t using it. And I guess if you are reading this then you probably know about it already, so I won’t go into too much detail on the basics.
10:12am: A shout out from Dale and Don. Thanks guys. Hi right back at you. Now they are talking FME Store, using an image that looks familiar. I think it came from my post on the store here.
10:08am: On the FME Desktop GUI, Don and Dale are now discussing the integrated parameter/attribute updates. Again, I’ve blogged on that before. This technology was proposed and created by a developer named Shelley and in France they actually call the new functionality the “Paramétrage de Shelley”!
10:02am: Did you know FME2012 now has the ability to create Concave Hulls. It’s been merged with Convex into a single HullCreator/Replacer. A setting for alpha defines the degree to which a concave hull clings to the outline of the data, but setting it is a bit of an art. For me, it works best when you are dealing with a set of scattered points.
9:56am: InlineQuerier. This is really cool. It’s a new transformer to build a temporary database mid-translation. It may be that we upgrade to a spatial database in the future. If you want to know more it seems to be one of the options for a lightning talk later today - so go vote for it. And of course I blogged about it here. It allows you to replace multiple FeatureMerger transformers by merging everything into the database, and extracting it out with SQL:
9:51am: Current demo is taking a raster DEM and using the RasterToPolygonCoercer (new for 2012) to create polygons of different elevations. Then he uses the Generalizer to smooth the polygons (as NURBS). Even I didn’t know we could do that. The output now looks like this:
9:44am: On Twitter I’m asked if we are recording the presentations for later viewing. I’m told we are. Keep an eye on our web site early next week. Meanwhile Don states: “If you’re excited about something, then just do it” - in reference to a Lidar slicing/profile example Dale is thinking of showing.
9:41am: Dale announcing support for Blue Marble reprojection. Official release is May 1st. There will be a Blue Marble reprojector transformer in FME.
9:37am: Working on a Lidar example. Lidar files are joined together, clipped to a site boundary, and colourized using cells from a raster than is overlaid on top. The results are also turned into a TIN and written to a 3D PDF. Here’s the point cloud output. It’s hard to see, but this is really just a cloud of points coloured using raster cells:
9:32am: A very good point Dale. With FME2012-SP2, you can now use SQL statements against a Google Fusion Table.
9:30am: Dale is running through an example of loading John Snow’s classic Cholera data into Google Fusion Tables. Did you know his data was crowdsourced on Ye Olde Faycebooke? You can find this same example in a recording of our Fusion Tables webinar at http://www.safe.com/learning/webinars/
9:25am: FME now supports 278 formats. Which is your favourite? New formats for 2012:
CAT (Column Aligned Text), AIXM 5.1, ArcPAD, AIS, LAZ, Oracle Point Clouds, Supermap, DirectX, CityGML 1.1, Google Fusion Tables, Free File Geodatabase API, OpenFlight.
9:21am: Proof that FME2012 is faster - “faster than a helicopter”
9:16am: Don and Dale are starting their review of FME and what’s new in FME2012. Did you know that the splashscreen for 2012 varies. There are four - three of which are randomly inserted. They just couldn’t decide which to use.
9:13am: Hi folks. I’m going to try something new today. I will live blog the FME World Tour event that is going on in Vancouver - and which you can follow live online. Simply visit this page and refresh it at regular intervals to get the latest going-ons.