Why d3 Will Change How We Publish Maps

Publishing maps to the web. Simple right? Just reproject all your data to web mercator and be done with it. Have we all learned to live with the crazy distortion? Do we need mercator to help us get our galleons from the Spanish Main back to the king? Of course not, but we’ve all learned the world is not flat since then.

Bill Gates

Five years ago I complained about polar projections because of a project I was working on. I even offered up a SVG example from Esri’s ArcWeb project (that’s how desperate I was!). What I ended up doing is using web mercator and then “transitioning” to Bipolar Oblique Conic Conformal as needed. It was ugly but the client was happy.

Well I’ve never really abandoned this quest. SVG has always seemed to hold a promise for client side projections of data, but support has been weak. This fall a new project has really gotten my attention, D3.

D3.js is a JavaScript library for manipulating documents based on data. D3 helps you bring data to life using HTML, SVG and CSS. D3’s emphasis on web standards gives you the full capabilities of modern browsers without tying yourself to a proprietary framework, combining powerful visualization components and a data-driven approach to DOM manipulation.

Bingo, D3 takes the browser out of the equation so you can just focus on the visualization. The really cool thing about D3 is that it gives you geographic projections. You can even add your own custom projections. Finally a library that gives you the ability to publish the data the way you want, rather than shoehorning it into some crazy projection named after a guy who probably thought there were sea monsters.

The examples are great and this sick D3 satellite view from @vtcraghead

D3 Sat

This isn’t the road network you are looking for

I have to say, gis.stackexchange.com is always an interesting read. You see questions from just about every type of location data user; from GIS Professionals to neogeographers (do we still use that term?). Yesterday there was a question I hadn’t seen in a while:

Is there any site from where I can get Google Road network as shapefile? As per my knowledge It is not distributed by google. So anybody can suggest how can I create this road network for a custom use?

So we all know that Google’s road data is proprietary and you can’t use it outside their APIs. I said this last month:

Clearly, we need to approach OSM differently than just a free slippy map. The value of the data created is immense and we’ve tried to make it as easy as possible to bring that great data source into your GIS applications. When I think of OSM, I think of possibly the greatest mapping resource ever conceived. The depth and breath of OSM still amazes me, even after all these years of being involved with the project. WeoGeo hopes that by providing free and customizable options for using OSM, you’ll be able to not only use OSM data in your projects, but also give back to the OSM community.

Clearly wise beyond my years, but it is time to stop using closed APIs such as Google’s or Apple’s and embrace OpenStreetMap. We’ve got the whole OSM world up on our market and I can clip out the USA, request only the highway data and deliver as a shapefile in a matter of minutes.


It’s that simple to get the OSM road network at as shapefile. Plus you can use it how you see fit, as long as you credit back the project. Hey, Merry Christmas Devils Dream!

OpenStreetMap Planet

OGC Draft Geopackage Specification

Well look at this, my switch to SpatiaLite might be short lived. Well not entirely because it looks like SpatiaLite is being integrated into a new OGC draft specification, Geopackage.

OGC Plenary GeoPkg_SWG 2013-1-14http://www.scribd.com/embeds/117538817/content?start_page=1&view_mode=scroll&access_key=key-1ar1w1ko565fokk8uj35

I’m the first one to complain about OGC standards (WxS, GML, CityGML) but this one might be something I can get behind. The specs are right what I’m looking for:

When I first saw it and the words OGC tied to it, I tried to hate it, I really did. But there is much here to like. One thing I’d like see added is support for cartography but maybe tiles is all people really care about.

Close Look

I really did try to hate this but a closer look and I’m in love!

The KML Problem

KML is if nothing well supported by many applications. We allow export of it at WeoGeo. But why is it every time I use it I get just a little bit angry? Take this simple OGR command:

ogr2ogr -where "description LIKE '%chair_lift%'" -f KML chair_lifts.kml output.kml "aerialway_line"

See that LIKE? It’s in there because the KML has no database. I’m basically searching the description bubble and finding something. I felt pretty dirty after resorting to that hack. Of course you COULD create a KML that had more “fields” but those are few and far between. It might be my database background, but I almost require a database backend for me to actually use spatial files. So how do I interchange GIS?

  1. Shapefile – No, I already said I’m leaving.
  2. KML – No database, no bueno (can I have a t-shirt with that?)
  3. File Geodatabase – Non-standard outside of the Esri stack
  4. WKT – in an alternate timeline we are happy and this is our interchange format.

I could go on, but you get the point. GIS formats are either poor choices from a technological standpoint or they are poorly supported. I’ve joked around quite a bit about SpatiaLite being poorly supported, but maybe it is time for me to get back on that wagon. Consider this my goal for 2013, use SpatiaLite as my personal GIS file format of choice.

GeoMonkey SpatiaLite

The GeoMonkey is wearing his SpatiaLite jacket waiting for the bandwagon to show up.

Rethinking the GIS GUI

While working on a project late the other night, I was thinking about how basically all GIS software is the same:

I mean everything is bad. ArcGIS, MapInfo, QGIS, TileMill, Intergraph, Manifold and any other you wish to list are all a complete disaster. We are able to use them because we are familiar with their workflows, but at what cost to productivity? It is a sad state of affairs when I’m excited to use shp2pgsql. I’ve noticed that I’m using GDAL/OGR much more lately and I think it is because it makes me much more productive. Says Bill Dollins:

Bill Dollins Tweet
Now I remember something Scott Morehouse once said,

[ArcGIS] is scientific software

I don’t recall the exact context of the statement, just that it has stuck with me all these years. The more I think about it, the more I think he’s right. GIS is complicated and all over the place. Workflows are if anything, unique to each user. That’s what I think makes the command line so perfect, I have to create my workflow from scratch.

But that’s a cop-out. Scientific software doesn’t have to be hard. I’m not a math wiz by any means, but software such as Mathematica and MATLAT are easy enough for me use without reading a manual or take a class. Try that with ArcGIS or QGIS!

I was just about to write about how the GUI is bad because Esri got here first and screwed it up, but that’s too simple. Part of it is that the GIS software space is mature and stable. What I mean by this is that you don’t see new competitors releasing products and being disruptive. And since the software isn’t retail, there is no worry about the PC World review. Basically you create a wall around your software product and make it hard to move to another platform. Every GIS program falls into this trap.

If North Korea can launch a satellite, why can’t we have a good UI for GIS software?

So for me, I’m sticking to command line tools, proprietary or open source, to get my job done. You better understand how tools work when you use them one on one rather than some wizard. I don’t really know if GIS software can be fixed at this point. My suggestion is they keep allowing us to script with Python and stay out of our way. That’s a win/win in my book.

OpenTripPlanner Team Navigates Away from OpenPlans

Update I didn’t know today was going to be blog about Transit App for iOS 6 day but AllThingsD has some more info on the situation.

One Kickstarter project I backed was Transit App for iOS 6 from the OpenPlans team.

Basically Apple punted Transit directions for reasons I can’t understand. Getting transit feeds for a company like Apple wouldn’t be hard at all. But that’s all water under the bridge I guess. When using Apple’s Maps application and you want transit directions, you get the following screen:

Apple Transit Options

Basically if I click the route button on those top two apps and you get transit navigation. It’s an extra step nobody wants, but at least it works. OpenTripPlanner was supposed to be an option for iOS users wanting transit. But then something happened:

Despite the successes so far, we’re stopping work on Joyride, and refunding you. Here’s why: the trip planning in Joyride depends on up to date, accurate transit data, which we assemble from open data feeds provided by hundreds of individual transit operators. In the new year, the OpenTripPlanner team is spinning out from OpenPlans. With their departure, we won’t have the capacity here to keep the app up to date, and we’re not prepared to launch without being certain we can offer a viable ongoing service that people can depend on for their daily travel needs.

Not sure what happened, the code is available on Github so I guess that’s something. OpenPlans suggests The Transit App but it proves that open data feeds aren’t a cure all (screw you ValleyMetro).

OpenTransit es no Bueno

The One Where Di-Ann Says Crowdsourcing Makes the World More Efficient

This week we had a great guest in Di-Ann Eisnor of Waze who was kind enough to join us right off stage from her talk at APPNATION. It was a great conversation about why Waze did what they did, how they’ve evolved, how crowdsourcing is improving their map and how they see advertisers as the way to make their map better for their users (very interesting). Di-Ann was on a WiFi at Moscone Center so the connection wasn’t great, but it is worth watching (it did clear up a bit later in the show).

You can watch the video anytime on the WeoGeo Video page (as well as the IRC log with a lot of questions and discussions) or see an archive of all the previous shows.

ogr2ogr every Shapefile

Earlier this week I was working on a map in TileMill and ran into an issue with a layer. At the time I couldn’t figure out which layer but Dane was good enough to help track it down. It ended up being an issue with the shapefile spec being vague and the Natural Earth shapefile being problematic. Mapnik and thus TileMill were not at fault, but the shapefile itself.

Shapefiles are used because they basically are the interchange file for GIS information. Others have tried with SpatiaLite and File Geodatabases, but neither has been accepted at large by the community. After spending half a day debugging my Carto for no reason, I’ve come to the conclusion that you must run ogr2ogr on every shapefile you share and every shapefile you consume.

Freaking Sweet


Assume every Shapefile is corrupt and your life will be much easier. I’m going to move to KML personally.


Or as a very wise man puts it: