Hobu, Inc.

Radiohead’s “House of Cards” video was done with LiDAR (no cameras).  Check out the Google code page with videos (and the data) describing how it was done.

Upload 4500 Oracle SRIDs to http://spatialreference.org, even though sr.org can’t interpret them.  Your IP address was XXX.XXX.XXX.XXX.  I admire your curl and bash skills, but don’t you think that maybe sr.org should provide Oracle SRID support through another mechanism than just bulk uploading them?  Please send an email next time…

/me wonders about adding more restrictions to who can post SRSs to sr.org…

Update: ok, the person who did this contacted me and I may have overreacted just a little :)  The reason why Oracle SRIDs aren’t supported by sr.org is they aren’t supported by GDAL/Proj.4.  Oracle has different names for many of its SRS WKT elements, they are different between versions of Oracle, and GDAL doesn’t have a dictionary to map these names to OGC/ESRI WKT names.  It is hoped that once CS-Map is released, we’ll have a dictionary to do this mapping, but until that time, open source GIS support for Oracle SRS WKT is pretty limited.

Some may have noticed hobu.biz had a week long outage a couple of weeks ago (not that I post enough for many to notice anymore).  This outage was related to the fact that the building that houses this server in Cedar Rapids had 10ft of water in the first floor for five days and the backup generator for the machine room had its wiring plumbed through the basement — preventing its usage.  Up until the 1000 year flood, my ISP had performed reasonably well except for the fact that they are rather expensive … $250/mo for 2U and 1.5mb of bandwidth.  I could have probably gotten a much cheaper colo someplace else, but it was the only thing within reasonable driving distance to Iowa City, so I took it at the time.

My ISP didn’t really handle the flood too well, and it was almost a week after the power was pulled on the machine before I was contacted.  A 1000 year flood was definitely not something they had planned for in their five 9’s ISO planning, and while understandable, me and my clients found it unacceptable.  Besides an unclean shutdown, hobu.biz is still diesel-powered and will be for the foreseeable future.  With the situation fluid and unstable, I started looking for alternatives…

I had been watching Amazon’s developer services for a while, and with the flood I’ve had strong motivation to investigate more fully.  I found the offering to be quite impressive, and I think I will be moving all of my essential services and websites to EC2 once I get an image built and my 3TB of data migrated.  I think EC2 and its competitors are really going to put the squeeze on the mom and pop ISPs with a 42U rack and a DS3 line.   

Are you running stuff on EC2?  GIS stuff?  What have your experiences been?

libLAS continues toward the march toward a 1.0 final release with Friday’s 1.0.0b1 release.  libLAS is a C/C++/Python library for processing ASPRS LAS LiDAR data, and it supports the LAS 1.0 and 1.1 formats.  It will probably support the upcoming 1.2 format as well once that spec is ratified.

With our approach of 1.0, we are looking for more beta testers to squeeze out any of the remaining showstoppers that we might have.  If you work with LiDAR data at all, give it a spin.

We just released version 0.9.3 of libLAS and it now includes ctypes Python bindings.  See the tests for examples how to use them, and head to the wiki page to grab a copy of libLAS if you’re into Python and LiDAR data.  (Note: libLAS also contains C and C++ APIs, and the ctypes Python bindings use its C API).

In the midst of having a baby (well my wife, anyway ), I was diligently working on completing OGR ArcSDE write support for the upcoming GDAL 1.6 release. This post describes the state of that work and asks for testers to come in and start giving it a whirl.

MapServer has supported querying from versions for almost four years (I think this is something ArcIMS still doesn’t support, but ArcGIS server does), but OGR’s ArcSDE driver only supported querying from SDE.DEFAULT and basic query operations. I was contracted to flesh out the rest of the driver, including write support and participating in ArcSDE’s versioned editing machinery. In January, I received an excellent patch from Shawn Gervais of project10.net implementing geometry conversion and basic ArcSDE layer/attribute field creation that was a fantastic starting point for the work. With the hard bits done, I was able to jump into the miserable bits — implementing the versioned editing/query support.

To be an effective player with ArcSDE, client software really must participate in the versioned editing scheme. After the “base” tables are created and loaded in ArcSDE for a multiversioned layer (”registered with geodatabase” in Arc* parlance), ESRI clients typically edit layers by manipulating the adds/deletes/modifieds tables through ArcSDE’s edit state/versioning machinery. If you would want your edits to an ArcSDE layer from OGR to show up in ArcMap, for example, your changes must happen within this machinery, or they would not be available until the adds/deletes/modifieds tables were “compressed” down into the base tables — an operation that requires locking out all clients from the entire database. The simplistic description of these operations is:

1. The version client wishes to edit on is locked, preventing other clients from moving its state
2. A new edit state is created
3. ArcSDE is told the edits within this session are happening on the newly created state
4. Edits are made
5. The transaction is commtted and the version’s state is moved to the newly created state
6. The lock on the version is removed

OGR doesn’t provide a clean mapping of a number of the concepts of ArcSDE, however. A first hurdle is transaction support — OGR has rather anemic transaction support that happens at the Layer level and ArcSDE’s happens at the connection/DataSource level. To overcome this, OGR’s driver puts all of the operations within the opening/closing of the OGR DataSource. I’m not sure I like this compromise, but I can’t think of a cleaner way to do it without foisting the pain of the user having to know what stage in the editing scheme a particular operation is at and how to roll themselves back from it. The consequence of this choice is that the editing state machinery is switched on as soon as an ArcSDE DataSource is opened for update and switched off when the DataSource is Destroy()’d. Re-opening an ArcSDE DataSource connection is rather heavy (1-2 seconds), so it might be necessary to revisit this choice if people want high-throughput, multi-user operations.

Another challenge is ArcSDE’s coordinate spaces. ArcSDE has fixed coordinate spaces that are defined at layer creation time. This restriction has to do with the underlying implementation actually using integers with offsets to store coordinate info. Coming up with reasonable defaults for defining this space is challenging, especially if you don’t know the geographic extents to which someone might attempt to store data. This problem isn’t limited to just OGR, however, and many an Arc user has had trouble coming up with good X/Y domain values. For OGR, the values are set based on the coordinate system, and there currently isn’t a way to override that default, but if people have trouble with it, it shouldn’t be too difficult to allow a way to override it.

The final hitch is that there are so many options with ArcSDE that don’t map to OGR concepts. Beyond the X/Y domain stuff, there’s whether or not to use multiversion tables, DBTUNE keywords, and so on. Many of these are available as either layer creation options or general OGR environment variables. More will likely have to be added as actual users start attempting to do real work with the code. Leaky Abstractions, indeed.

Come help test

A final note for folks who might be interested in this stuff. Post a comment here or email me if you wish to get a Windows build (against 9.1, 9.2, or 9.3 (beta program FTW)) to test things out and see if they behave properly for you. I have an EDN license, which gets me ArcSDE, but it doesn’t get me ArcMap, so I have no way to really test if things are behaving exactly as they should. Testing help would be much appreciated.

Thomas Ronald Butler. March 19th, 2008 @ 4:15 CDT. 6 lb. 0 oz. 19″.

Quite unbelievable.

Could something like this happen with open source software?  Absolutely, but I wouldn’t have to use a .NET decompiler to find out.

One of the newer Open Source GIS technologies I’ve been playing with lately is GeoDjango, and I really like where it’s going. While I’m barely passable as a web developer, I do have a lot of experience with Zope/Plone, and GeoDjango fills some big holes in the Zope stack when it comes to geospatial stuff that I don’t see ever being filled. In this weblog post, I’ll give you a little introduction to GeoDjango and why you might be interested in taking a look at it.

GeoDjango is Justin Bronn and Travis Pinney’s effort to geoify the Django project. Django has been gaining a lot of momentum in the Python space, with ex Zope Corp employee Guido Van Rossum pronouncing it as *the* web framework for Python, and high profile deployments like EveryBlock attracting attention has given folks an excuse to take a look at it. While every great web framework comes with great compromise, Django does a number of things right, and GeoDjango takes things even further. In addition, GeoDjango expects to gain quite a bit of profile in the next few months, especially with its prominent placement at Where 2.0, and it’s one bandwagon that I’ve been happily jumping on.

What does GeoDjango offer that isn’t available in other frameworks?

GeoDjango’s power comes from its ORM that helps you abstract away the (geo) database. It doesn’t take the DB entirely away, and there are some limitations, but it makes for an excellent 80/20 solution. When combined with the development momentum sweeping through the rest of Django, it provides a developer with a lot of power. But, as they say in the informercial, that’s not all…

Another advantage of an approach like GeoDjango is you *still* have your database. A distinct disadvantage of Zope is that you are pretty much married to the ZODB. While there are options to be able to pull stuff from something like PostGIS, it’s neither the preferred nor popular approach — you’re on your own. Most of the other applications in the open source geo space can all speak to PostGIS, but hardly any can speak to ZODB. This means you are out in the cold if you want to use other applications (say MapServer for fast rendering or QGIS for desktop visualization) in the spaces that they excel at.

Finally, the nature of Django’s views practically forces you to take a RESTian view of the web development world. My first experience with this was working with Chris on the development of spatialreference.org. I had been playing with Django, and it was natural to have the distinct SRS output types as URL-based resources. This connection is what precipitated the website’s development.

Some areas where GeoDjango could stand to improve

GeoDjango has a health dose of NIH syndrome. It maintains its own custom ctypes bindings to GEOS and GDAL instead of leveraging existing work in those areas (Shapely and my official GDAL Python bindings). Short term, I can’t argue that this approach has really hurt the project, but longer term, duplicate efforts like this have a chance of leaving GeoDjango lagging as the efforts with more developers and momentum continue to march on. I think it would be a long term benefit to combine these efforts where possible.

There are a number of work items that need to be done to merge the GeoDjango stuff into the main Django branch. GeoDjango exists as its own little world right now, and it doesn’t track Django exactly. I think this issue is just a manpower one, but it will be great to have geo stuff be first class in Django.

Mateusz and I have been diligently working on a new open source (BSD licensed) library for reading the venerable LAS LiDAR format in C/C++. The library, called libLAS, exists to provide stream-based access to LAS 1.0/1.1 files. It builds on work by Martin Isenburg, but it attacks the problem with a software engineering tact, rather than aspiring to be functional academic software. Martin’s library, for example, just printed all of its errors to stderr, and it was expected to be used in the context of command line utilities for the most part — not embedded as a library in other software.

It is expected that we will eventually hook libLAS up to something like OGR to provide access to LiDAR data in the context of a GIS processing library. LAS data is typically very big (millions of points), so there will definitely be some challenges to overcome to make it useful in a library like OGR, but it should be useful for straight-ahead translation operations. There would be nothing preventing you from using libLAS in your own software development, however, and Mateusz and I are looking for collaborators to help polish the library and cover the usages that most people would want. Consider this weblog post as your call to participation!