ESRI Binding Data.gov and GeoData.gov
June 2, 2010 27 Comments
Bring in a Middle Man
Late last week, the following news came out.
The White House has contracted with a major developer of mapping software to merge a federal website that publishes geospatial information with Data.gov, the government’s depot for downloadable data sets, the company’s president said on Thursday.
California-based ESRI began last summer tying Data.gov to Geodata.gov, the geospatial information gateway, said company President Jack Dangermond in an interview with Nextgov.
So lets get the “bottom line” out of the way first. Clearly ESRI is doing this at practically zero cost for the feds. When I first read this I couldn’t help but chuckle. Competing against Google, who is more than happy to give products away, forces companies to make hard decisions such as this. I can see Mr. Kundra sitting in his office laughing uncontrollably because he can play Google, Microsoft and ESRI off each other to get what he wants for practically free. There are very few geospatial companies that can pull this off at that price and honestly ESRI might be the only one.
The GeoMonkey enjoys geo-content only one way....
Visitors to the Feds or Visitors to ESRI?
The NextGov article is somewhat vague, but leaves a bad taste in my mouth.
The site [ArcGIS.com] already allows anyone to search for graphic layers of information from data sets ESRI retrieved from federal GIS databases. Visitors then can add the layers to a base map, or a background map, to complete the picture. ESRI, which makes money by licensing software for managing and publishing geographic information, is offering the site free of advertisements and does not claim ownership of any content that people and agencies contribute, Dangermond said.
How am I to read that? Does this mean that because I consume “open” federal data in ArcGIS, I can access it for free without advertisements (take that Google BTW)? Must I use ESRI APIs to access geospatial data in Data.gov? It sure reads that way doesn’t it?
You'll use ArcGIS.com! Now leave before I am forced to taunt you a second time!
Ah, but the scuttlebutt is that this was a quick sole-sourced contract that is meant to drive content to ArcGIS.com. I guess if someone came up to me and said they’d do millions of dollars of work for free, wouldn’t you take them up on that? So that makes this about people going to ArcGIS.com and not going to the EPA, or DHS or whatever 3 letter agency you enjoy working with. ESRI won’t give away something unless they get something in return. That would be ArcGIS.com being the middle man.
Open vs. Open
All part of the master plan...
The other big issue I see here is the question, “Is ArcGIS.com open data?”. Do “open” ArcGIS Web Services make this data open to all? Does data in ESRI Layer Packages or Layer Files equate to open data? Hard to say. I asked that question on Twitter last week and got 100 different responses. To be honest, I have trouble answering the question as well. I just got back from the NYS Geospatial Summit where I said that I get very nervous when government data is passed to me through proprietary APIs, but as long as I can get at the raw data in an open standard I’m OK with it. Thus I think as long as ArcGIS.com is there for web services and I’ll still be able to get at the data through agencies websites (e.g. EPA.gov and not ArcGIS.com) there isn’t anything to worry about. But I don’t think that is the case at it appears to me that everything will be run through ESRI’s servers and the data will be available on ESRI support platforms.
The Playing Field
As I said above, we’ve gotten into a vicious marketing cycle here. ESRI can afford to do this because they can sell their licenses to the Federal government to produce the services. Could anyone else in our space do that? Nope, not a one. But what does this mean for consumers of federal data? Does it mean that we all need to buy ESRI licenses to work with their data? Possibly, “open” RESTful services or not, the bottom line is that those with ArcGIS Server or Desktop in their pocket will be able to get up and running quicker than those who have Autodesk or Intergraph licenses. It puts pressure on users to come up with ETL solutions (like Safe’s FME) to do the conversion for them, from ESRI formats and services to OGC formats and services.
A Funny Thing…
What this is all building to though is actually a pretty usable system. Can you imagine, assuming Jack can pull this off and get all governments (local, state and federal) to load their data into ESRI’s ArcGIS.com, how valuable this might be? Ignoring the “open data” question for a moment, if I can overlay data from my (well maybe not Tempe,AZ) local city, my state and federal government agencies all in one place; doesn’t that give us the tool we all want? Funny how we get right up to the door and all we have to do is walk through. We’d then enjoy access to thousands of geospatial datasets. Maybe my problem is that there are locks on that door and I’m just afraid that I won’t always have a key to open it or get back out.
The ESRI UC is shaping up to be quite interesting, isn’t it?
About James Fee
The Onion: New Marlboro Earth eliminates the causes of global warming and environmental destruction at their source.
http://www.theonion.com/articles/new-ecofriendly-cigarettes-kill-destructive-human,17529/
This is not as unrelated as it may appear.
So I have to access public data trough Geography Network 2.0? Ugh…
I don’t see what the problem is, we’ve already sold our soul to Jack. Might as well just enjoy it.
Open ArcMap… File > Add Government GIS Data
Brilliant!
Sole sourced contracts are fun aren’t they? Jack and done the impossible, made ESRI the most valuable company the world has never heard of.
On the same front they “sponsor” open projects such as OpenStreetMap 52North and OGC to make it look like they care, but they don’t.
Lock in for ArcGIS Server and get the fanboys geeked up (http://twitter.com/dbouwman/status/15229841683). Yep, mission accomplished!
So, anyone actually try to find anything useful at geodata.gov yet?
I cannot even find our own data sources and applications that I know are in there. Far too many nationwide datasets that come up as the first 800 or so hits when trying to find local datasets.
I’m sure you didn’t use “useful” and “geodata.gov” in the same sentence.
Geodata.gov is a waste of space. I can’t imagine how ArcGIS.com won’t be better than it.
I’ve never been able to use Geodata.gov for anything other than as an example of what NOT to do.
As I presume you have seen quite many different solutions for data sharing it would be intresting to hear wich ones you like and perhaps – as Sean writes further down – have them as a sort of blueprint for building a better solution.
1) Load your data into GeoServer
2) Develop a simple OpenLayers map interface
3) Expose the many output formats of GeoServer
This solution, based entirely on open source components works well for St. Lawrence County, NY: http://www.opengeohost.com/maps/stlawrence/
Basic users have an intuitive, interactive map for viewing and querying the data. GIS “Analysts” can download the data in various open formats. Application developers can consume the web services (WMS, WFS, KML) and make their mashups.
Great, it is hard enough to get data from the feds. Just another stupid step between me and public domain data. Why go to this extreme when all we need is a fucking FTP server?
[Sigh]
It was recently clarified for me that geodata.gov and data.gov do not actually store/serve data, but are aggregators of the myriad of sites (presumed NSDI nodes…’member that effort?) where feds actually store/serve data.
As for the usability of geodata.gov, it appears to be designed after the now DOA Geography Network, so a redesign/assimilation into data.gov is warrented.
The sketchy bit here in my mind is how closed the whole deal was. It was a sole source extension to ESRI’s 2004 contract for GeoData.gov. We can all agree that GeoData.gov was a metadata repository not a data repository. So, why would a six year old contract be justification to build in geospatial data support for Data.gov? I hope someone in the watchdog community is paying attention to all this.
At the end of the day neither the government or the incumbent vendors they hire have done a great job at any of this. That was the whole supposition behind open data. To put the data out to the community to build innovative applications with it.
If ever there was a time for the open data community to band together this would be it. Build a better mouse trap then share the data across the community in open formats. Ideally make it federated through Opensearch and Atom.
http://blog.fortiusone.com/2010/01/05/better-know-a-geocommons-feature-opensearch/
The best way to demonstrate something is wrong is to do it right. We all demonstrate how to do it the right way and then the government good guys can point to how data should opened up and shared. Anyone who wants to interconnect data just let us know.
No discrimination here – happy to pump it right back into your ESRI apps:
http://geobabble.wordpress.com/2010/06/02/importing-data-from-geocommons-into-arcmap/
Thanks to Bill Dollins!
The point ESRI seems to miss is we are all part of an ecosystem called the Web, and it is not a mutually exclusive desktop game anymore.
Sean,
I don’t think ESRI has missed your last point at all. I think they’re completely aware of (and dead-ass scared of) the web.
So they wangle a $50k (fifty thousand!!!) hush-hush contract extension to get the work done and capture the data for themselves.
Consider them a caged animal fighting for its life.
Yuck, is it time to move to Canada? What happened to open data? George Bush II had more open data than Obama does.
Pingback: Why ESRI’s GeoData.gov - Data.gov Contract Gets the Community Upset | Off the Map - Official Blog of FortiusOne
I agree with Sean. Please try to keep your data free and open, and hopefully not locked into ArcGIS.com!
Make it easy to find and freely available for download as shapefiles, or available for developers via GeoRSS, WMS, WFS, KML, and GeoJSON data feeds and REST services. In addition to Geoserver’s many output formats, Naniamo has done some good work in this vein too using GeoRest.
Indeed, see my latest blogspot on same
http://bit.ly/ayAwo2
You need Safari to view Apple’s HTML5 showcase. Same concept.
http://www.apple.com/html5/
As a company, I think ESRI did a right thing. In addition, your federal gov is more open than our Chinese gov. The data here is a secret.
By the way, Can I translate the article into Chinese?
Pingback: Why governments should outsource open data to ESRI and Google « Georeferenced: Thierry's Blog
Marten Hogeweg said it best in his blog
http://martenhogeweg.blogspot.com/2010/06/building-your-own-arcgiscom-client.html
“ArcGIS.com provides a great collection of resources and, as Jack explains below, allows other people to discover the work ESRI users are doing.”
Emphasis on “ESRI users” this is not meant to be open…
They do provide an open CSW search service, but to consume you need ArcGIS APIs or clients.
From clarifications on this blog and Marten’s other post that same weekend, let me rephrase the data.guv issue not so much openness as access (see my previous comment above)
Can any anyone say:
THE FREEDOM OF INFORMATION ACT
5 U.S.C. 552
As Amended by Pub. L. No. 104-231
(ii) disclosure of the information is in the public interest because it is likely to contribute significantly to public understanding of the operations or activities of the government and is NOT PRIMARILY IN THE COMMERCIAL INTEREST OF THE REQUESTER.
The section you quote refers to “the schedule of fees applicable to the processing of requests under this section and establishing procedures and guidelines for determining when such fees should be waived or reduced” and is thus hardly applicable here.
http://www.justice.gov/oip/foia_updates/Vol_XVII_4/page2.htm
Correct…and the deal stinks, regardless.
Still, it’s better than nothing. Sigh.
When did our criteria for success in government become something/anything is better than the nothing we’d get otherwise? Never let it be said our government did anything more than the least they could do.