Sharing Cartography
April 21, 2009 62 Comments
There are tons of ways to store spatial data, but sharing cartography is much harder. ESRI seems to be using their new Layer Package “format” to keep the ESRI layer file with the datasets. Outside of that I don’t really see much in the way for cartography support with data layers (other than the odd ESRI LYR file offered up on FTP sites). In a way, there is much to like about ESRI’s decision to use essentially a sidecar file for storing symbology and maybe there is a more open method of sharing this information.
Most of the time when you download data, you just get the raw file format (usually shapefiles). You then drag your new downloaded dataset to your viewer (possibly ArcView, maybe uDig) and then you proceed to try and figure out how to color the map up (of course there is no metadata so you can’t figure out what field you should symbolize on) in an attempt to match that PDF of a scanned map that someone gave you. Wouldn’t it be better if there was included symbology that allowed you to view that data layer the way the author intended?
Well as I said, ESRI’s Layer Package format does this. The Layer file is of course a problem because you need ArcObjects to read it. But in an ESRI world, that isn’t too much of a problem. Outside of an ESRI world, maybe the MapServer Mapfile might be the best way to get symbology information in (heck you can write out a Mapfile from ESRI, let alone all those open source clients). Some have said KML is that format. I disagree because it doesn’t lend itself to storing attribute information (maybe there are ways, but I still question if KML should be used this way). At least KML does store simple symbology with the dataset.
Ron Lake offers up SLD/SE as an option, but acknowleges that it really has no traction. I really haven’t seen SLD used much beyond some GeoServer documentation and I’ve not seen anyone try and use SLD for storing cartography in desktop clients (NOTE: According to comments uDig does support SLD, but not be default which is a good starting point). Sean Gillies thinks CSS is the best option but again we run into the issues of desktop clients not supporting it and probably won’t (though Sean gets GeoMonkey cred for continuing to use the term “GeoWeb” every chance he gets).
What about Mapnik? It does a superb job at rendering great maps and Andrew Turner says there is a project that can convert CSS into Mapnik. Now that might be a workflow. CSS is very easy to edit and and work with and Mapnik is great at rendering cartography (and could easily be supported by desktop clients). CSS gets stored as a sidecar file to the dataset and the rich cartography can be read with Mapnik to render the maps as the author wishes the maps to look. Maybe Sean is on to something after all…
If we are going to go out on the GeoWeb wall of death, we'll need a sidecar.
About James Fee
I am not quite sure how CSS would work for more complex symbols. Also, it does not solve the need to drive your symbology by database values etc. But to be honest, I haven’t looked into Mapnik and its approach to these issues; I definitely intend to do that ASAP. ESRI’s lyr is of course a direct serialization of all the associated objects, which is after all very flexible and supports symbols, renderers and objects other which are not yet implemented or come from third-party developers.
But I also share the need to have a common standardized format for symbology. SLD is naturally very webservice-centric and for some reason, I have a strangely negative feeling about CSS being applied to map symbology. It just doesn’t seem right to me. Ideal would be to put all the positive aspects of SLD, CSS and possibly SVG together and have a single standard usable for cartography.
If we wanted to take this a step further, and now I am just thinking aloud.. we could also have a simple format to describe map page layouts, after all. Chances are there is really no demand for this, but I personally would find such a thing very useful.
CSS is certainly good enough to make websites look neater than the typical GIS map. And have you seen this? Case closed.
The case clearly isn’t closed Sean. Just because CSS may be able to do this well, doesn’t mean it has a hell of a chance of being implemented as such. You always focus on the technical side of things which is why I pay attention, but the reality of the situation means that CSS has zero chance of being supported as you propose.
I like the idea of a “sidecar”. The problem with KML has always been that I have to convert my data from its source to KML, losing lots of great info. I’d just assume leave the symbology on the side for those who want to use it. We’ve used LYR files in the past and while they work well for ArcView, we’d rather use an open standard.
Who is working on this James?
I’ve tried to use Mapnik in the past and it has been difficult to say the least. In fact I break out in sweat just typing its name.
I do like the look of Mapnik rendered maps, it wouldn’t be bad to have that quality in more desktop applications. The problem comes back to ESRI and they sure as hell won’t support this stuff unless it becomes an OGC standard (they they’ll have to support it). Thus we’ll end up with layer packages and then this open standard. I guess two types is not that bad.
Oh I love the CSS idea. Why create something from scratch when we already have a great standard. I do have to wonder about support as others have mentioned above. ESRI already has a “solution”. OGC already thinks they have one. The other players are too fragmented to join together to create an alternative.
Oh well, until someone creates an open source LYR render tool, we are probably stuck.
Of course Manifold already has this figured out. Their .map file stores everything in there. So if you’ll all buy Manifold, we’ll be set.
That’ll be the day
The SLD format is designed exactly for this. “Lack of implementation” is a chicken-egg argument: except for LYR there is no widely deployed implementation, so lets throw our hands up and go home. Or folks could pick an open standard and implement to it. You can see the .sld sidecar in action with uDig, which will use a .sld sidecar file to automatically style shape-files. This meme is what I was getting at last week: open data formats are just the tip of the iceberg for useful interoperability.
Another file to add (and easily loose, rename, destroy) to that mess of individual components that constitutes one shapefile
Why can’t anyone come up with a new interchange format, that is:
* robust (one file)
* no limitations (size, attributes … )
* open and documented
* supports all geom types
* stores cartography and metadata!
That would be the day!!
I like the CSS approach as well, take a look at this hydrologic data map. Look at the various parameters under ‘Select Hydromet data to display’. The symbology on the map and the coloring in the legend are controlled by a style sheet. Makes simultaneous updates to the legend and the symbology very easy.
YES! I’d love to use CSS to describe my symbology. We’ve struggled to share our data with Neogeographers because they don’t have access to ESRI software and thus cannot consume LYR files. Maybe the MapFile solution is something we should look into, but I’d rather have something more formal.
I suppose I’d be fine with SLD, but given how complicated every OGC standard I’ve ever dealt with is, I’d just assume not deal with it. Mapnik is an interesting project, maybe we can see some of that integrated into uDIG and QGIS?
Regarding using SLD in desktop clients…you mention uDig earlier in the blog post, but do not acknowledge that uDig supports SLD to style data. Of course you would need to download or otherwise acquire this SLD in addition to the data, which I believe is your exact point, that it should all be self-contained.
OK, then I stand corrected in regards to uDig. It was something I wasn’t familiar with.
So how do we build on uDig support of SLD?
Well since the shapefile is already a collection of files, if a “shapefile.sld” exists in addition “shapefile.shp”, etc. then perhaps automatically apply that SLD. I’m not a fan of adding non-standard extensions to what I presume to be actual standards (I’m not familiar with the history of the shapefile format and how standard it truly is), but this could certainly be help for uDig users and could gather some steam if it is done right.
This is already what happens. Give it a try. foo.shp, foo.shx, foo.dbf, foo.sld. BTW, re: GeoServer, we should have a web-based authoring tool which will generate SLD (natch) in upcoming releases.
So will uDig also generate a foo.sld if I create some symbology? Or is this a different process.
Oh James, that would be far too easy…
BTW, if you are trying to author an SLD for use in GeoServer, writing them in uDig’s GUI and applying them to the shapefile until you get the desired iteration is the only way to do it and maintain your sanity IMO.
Thanks,
Bruce
Here is what the udig online documentation has to say:
SLD import and export (from the style dialog)
SLD listed as one of the files that can be included with shapefile
Checking the complete list of where sld is used it appears you can also drag and drop an sld file onto any layer
The sample data set uses this to some effect; and although it is not particularly user-friendly you can directly edit your SLD file in uDig (people often use this to set things up interactively; and then export the result to geoserver).
We have an in-house developed desktop solution (standalone and integrated with ArcGIS) that acts as a metadata repository, together with symbology definition. The symbology is stored as ArcXML snippets at the moment. We accept the fact that that limits is to fairly simple symbology.
A user can search for the required data layers and load them into ArcMap or generate an axl to send to a colleague who can then create a map from it.
In the near future we are planning to port this application to a web-based solution and switch to SLD as the definition language. This aligns with a move from ArcIMS to Geoserver, primarily because ArcIMS is moving away from ArcXML to mxd based management.
I like the idea of using SLD’s for this. I think KML has its place, and is great for sending someone a self-contained presentation, but I would much rather have an SLD sidecar that is uniformly supported by GIS viewers. Yes, the OGC spec for SLD is long and complicated, but I have more confidence that it can do the job than extending CSS can. The fact that MapServer supports SLD through WMS requests, and that GeoServer uses it natively means that already there are 2 renderers that can (sorta) share the same styling info. It would be cool to look at a styled layer in GeoServer, export the layer in whatever format, and have it also come with the SLD. It would also be good if MapServer had the ability to use an SLD for layer styling internally as well.
I frankly don’t care what ESRI chooses to do with regards to this. They have had plenty of time to come up with an easier way for people to share styled GIS data with one another. The SLD seems to have a decent shot at doing a better job, and if we do more to adopt it, ESRI will end up supporting it anyhow. Just like KML.
“And if three people do it! Can you imagine three people walkin’ in, singin’ a bar of “Alice’s Restaurant” and walkin’ out? They may think it’s an organization!”
“And can you imagine fifty people a day? I said FIFTY people a day . . . walkin’ in, singin’ a bar of “Alice’s Restaurant” and walkin’ out? Friends, they may think it’s a MOVEMENT, and that’s what it is”
This hits the nail on the head. The requirement here is for something that can be used to represent map symbology, which can be easily processed and hand-written if necessary, and which has been specifically designed for the job in hand. That’s not to say that a CSS representation of it won’t be useful, but some people will also want a Flash, or SVG, or some other delivery format which we don’t have yet. SLD looks like a good candidate for a format in which to *specify* symbology, retaining the ability to *delivery* it in an appropriate format for the medium and audience.
note: not only uDig but also OpenJUMP and Kosmo have SLD support (implemented by Lat/Lon – i.e. the deegree people). And as far as I can see from a short google search: gvSIG has SLD support too. So..lots of FOS Desktop GIS Apps support it
When I looked it didn’t appear that they were reading SLD as a sidecar. Writing and reading SLD is only half the battle here IMO.
But I’m still not sold on SLD anyway.
call for participation:
gsf – geo sidecar file. a new standard for sharing cartography. please send your papers (more than 300 pages, one normative and one informative XSD structure and else) to gsf@opengeospatial.
I’m not familiar with SLD, but layer files are probably one of the most useful tools to come along in ArcGIS since the jump from 3.3
Layer files store a lot more than just symbology, pretty much every “property” of a data layer in an ArcMap data frame can be saved in a layer file.
This include theme definitions, joins/relates, labeling parameters, scale dependent display parameters, as well as connection information to source servers (WMS, ArcGIS, etc). Layer files also work with any data type that ArcGIS supports – raster, vector; shapefile, geodatabase, coverage; WMS, SDE, etc.
The main limitations (for me) are that the file is closed and binary (yuk!), and that data paths are hard coded into the .lyr file. As far as I know, file based data source locations cannot be made relative to a system variable. You need to write a custom (ArcObjects) data loading tool, test the data source, and redirect it as necessary prior to actually loading the layer into your data frame.
Even if you can get beyond the closed-proprietary nature of the system, this hard-coding of data sources makes data distribution and data sharing especially difficult. It’s the old “Where is…” dialog all over again.
This is the only issue that I believe ESRI is addressing with the new “bundle” concept at 9.3.1, however for ESRI only shops, it will open the doors to really taking advantage of the intelligence that a subject matter expert has put into the layer properties. Add the intelligence that such a person can put into the geodatabase as well and you have a very powerful combination.
None of it is open, it doesn’t do anything beyond the ESRI space, but if you wonder why this is something that is being touted at the various conferences, this is it.
Of course the big missing piece for the rest of the world is an API to the geodatabase, including the layer “file” component. I have been complaining to our ESRI reps about that for years. I work for a public agency and despite how much more convenient and time-saving it would be, I refuse to adopt the geodatabase as a public data distribution option until third party apps have a free API that they can use to access the data.
ESRI has supported relative path names in ArcGIS in Layer files for as long as I can remember.
http://webhelp.esri.com/arcgisdesktop/9.3/index.cfm?id=205&pid=197&topicname=Referencing_data_in_the_map
ArcCatalog always makes layer files with absolute paths. ArcMap will save a layer file with a relative path if directed to do so. You can change this setting in the Document Properties window.
Thanks for the pointer John, I’m glad to learn about that.
Is there a workflow for converting ESRI cartographic representations to SLD? It’s not very difficult to make some pretty awesome looking stuff with cartographic representation, and obviously the cartographic rules translate easily to SLD. (We’ve actually been using file geodatabases with cartographic representation rules to move data with symbology, but this has the whole ESRI limitation on it.)
Well ESRI already “supports” SLD so they could easily extend it to the desktop if they wanted.
http://webhelp.esri.com/arcgisdesktop/9.3/index.cfm?TopicName=Overview_of_OGC_and_ISO_support
I’m guessing that they’d rather just use LYR because it gives them freedom to do what they want. That isn’t to day someone couldn’t work something up for ESRI customers. The problem is that without ESRI support out of the box (writing SLD for data features), there won’t be much traction.
How about converting the *.avl file? It’s ESRI, text based and is automatically read with the shapefile.
Easy peasy, Bruce. Unfortunately nobody uses them anymore
ESRI still reads AVL at 9.3, but they don’t write them. Again writing SLD is the problem from ArcGIS.
All this is too complicated. Want to share a map with correct symbology. Hard copy it!! All the symbology stays with map, doesn’t change when the map changes hand.
Brilliant! Next problem?
O’, wait, y’all want digital…
I think they call that a PDF KoS!
I kind of have to agree with KoS here. If I want a map with someone else’s symbology, why not just get it as a hardcopy? Or as an image? The whole point to getting data as data is the ability to symbolize it to your own needs. I just don’t see a need to get data symbolized according to someone else’s standards. Am I missing something here?
If I download data from the US Fish and Wildlife Service, I’d rather use their symbology than my own. Sure I can take a JPG or PDF and determine the RGB, stroke and line types, but I’d rather just use their standards. Or what about FEMA data? Shouldn’t we be using a cartographic standard when using flood data? I’d hate to think everyone is doing their own thing (which of course is what everyone is doing).
You don’t have to use the LYR or even the SLD/CSS, but it is there if you need it.
Fair enough. I guess I’m just weird. I’d rather use my own symbology.
And you’d still be able to do so, but you’d have a sidecar file that you could leverage if you needed to. Many would ignore it, but at least there would be something for users to utilize in their mapping.
You could always override the accompanying symbology. I would love an open format that includes symbology. For the Planning Areas of the NJ State Plan, we’ve included a field with source information for parks in the Plan. We’ve then received maps from consultants with 10 shades of red and purple for each type of parkland because whomever made the map though it would be acceptable to just go with the default color palette in ArcMap.
With a default symbology, they would have the standard color values for the Plan Map. Then, if they wanted to make parks something other than green, they could.
“If I download data from the US Fish and Wildlife Service, I’d rather use their symbology than my own.”
Does anyone know if the USFWS or FEMA offer ESRI .style files to go with their data? If they have standard symbology then they could pass it off that way.
Don’t know about other agencies, but for FEMA and DHS, NIMS symbology already defines the standard. Problem is that NIMS is absolutely ugly
(even if every first responder knows exactly what it means)
And yes, they publish .style files to go with that symbol. Heck if I can remember where to get them though (other than off my own server).
There was a sample ArcObjects script under 9.0 that allowed you to store Feature Layers within a geodatabase.
http://edndoc.esri.com/arcobjects/9.0/samples/cartography/layers/databaselayers/databaselayers.htm
Now, if the File Geodatabase becomes an open standard, we would have a decent open storage format that could store symbology.
Great topic. But before someone develops an open standard symbology file perhaps there should be more concrete open GIS application. I feel like its all over the place and as more ArcMap people use the geodatabase we are more likely to have more confusion on what open data format symbologies are used since I feel like gov’t data will be offered more in the geodatabase format. Here at work, I have the non-ESRI licensed folks using QGIS but am torn when I receive great data in geodatabase format if I should backslide to .shp and have open usage or utilized the functionality of geodatabase with no distribution.
Open GIS just doesn’t offer the same graphic design as an ArcGIS -> Illustrator combo. Until QGIS offers export to AI I’ll continue using ArcMap.
One good thing with mapnik is its posibility to create SVG-file wich is both W3C standard and you can use in Adobe Illustrator or Inkscape. It shows that one can create a good graphic output with open source and I dont think it will take too long before this will be an option for applications as qgis or geoserver as well. Another intresting aspect is how to add cosmetic layers whith simplified symbolisation for different geometric option (such as replacing a building geometry with a dot in a sertain scale). This is possible in INtergraphs Geomedia and ArcGIS (I think) but I have not seen anything like this in open source.
I think you can specify SLD rules e.g. to be applied at certain zoom levels using the min/max scale denominator values to indicate when to apply the rule. Or depending on what you’re using to view the map, maybe you could simply swap SLD stylesheets as you zoom in/out. Kind of clunky though.
Still, as a non-GIS-expert, the idea of having a map (the model) separate from the styling definition (the view) strikes me as being more consistent with the general MVC approach in mainstream applications. So IMHO, SLD seems a better and more flexible solution in the long run – if better tools can be developed to support using it – than yet another proprietary format that mixes up your map data and styling in an impenetrable bundle.
But what do I know, eh?
I think you are right on ChrisW. Using MXDs to serve up maps is crazy. I know ESRI is moving toward MSD which “resolves” some of the issues, but wouldn’t it be much nicer to work it this way on both the Server and the Clients?
@NYGeog – the fact that ESRI is moving to a new, proprietary format is indeed quite concerning. All the more reason to work on a equally technically capable, and open format that we can all use and share data.
There are a couple of very simple possible solutions for sharing a styled vector data:
1) Shapefile + SLD
2) WFS + SLD
3) SQLite (via SpatiaLite) + Mapnik XML
4) Cascadenick (Mapnik CSS) + Shapefile/SQLite
5) KML
6) OpenLayers styled JS + WFS/GeoJSON
.
zip any of them up and call it .olz (open layer (package) zipped)
Then just convince someone to write an ESRI plugin to import/export these formats so they can join the Web as well.
Uh hello …
ArcMap2SLD
1
2
3
Authoring SLD isn’t the problem.
Getting ArcMap to actually use it as it would a LYR file is the issue.
So then have your ArcMap LYR as the canonical copy and export to SLD whenever you want to update your GeoWeb map. Or inlcude both the layer file and the SLD when shipping the shapefile.
That is still too ESRI-centric. Since SLD/SE is an OGC standard, there isn’t any reason why ESRI can’t support it on the desktop. If a non-ESRI customer (I know amazing that there are still people NOT using ESRI) wants to share symbology, they’ll have to rely on ESRI users ability to read that SLD.
Right now that isn’t possible.
Well go ahead and file a bug. If each ESRI customer, especially some of the larger ones, did this it would start to get attention. I mean the place to focus is pretty clear on this one. SLD is a standard – there are several projects that support it and one that doesn’t.
The main problem is getting style information in-to and out-of ESRI. Those .lyr files are pretty much useless with external software, and ESRI doesn’t do a great job with SDL.
One possibility is to import and export in “ArcPad” format. This is a single button operation in ArcGIS. It produces the standard shape files along with a .apl file which contains ArcXML style information.
If we had an SDL/APL converter, then we could move style information to and from ESRI with relative ease.
I guess I’d rather see ESRI give full support for APL, but in the meantime, an XML–>XML converter would be relatively straightforward to implement.
Maybe it would be more appropriate for ESRI to provide SLD compatibility in their Silverlight/WPF SDK. So far, I don’t see any tools for creating renderers for silverlight layers.
That way someone could author a renderer using Expression Blend, and save it as xaml along with a shapefile to a zip file. A silverlight app could then retrieve the zipfile and load the renderer using XamlReader.
Given that transparency of AGS layers is not supported, providing ways to pull symbolized data into the browser as a graphicslayer (which do support transparency) should be a provided.
You can use MapDotNet UX Studio to create a .mapx file which stores map and layer definitions as well as symbology in XAML. In also allows for arbitrary metadata (key-value pairs) at the map and layer level which you can access in your application. Studio is free and has an “export map” feature to share maps. Yes, it’s a commercial product. The file format is XML and XAML for symbology provides some flexibility for designers. For example you could create your point symbols in Microsoft Expression or convert vectors from Photoshop.
PDF files do a good job of rendering accurately the authors intent. What was called GeoPdf, now TerraGo Desktop claims cartographic consistency. They also claim some collaboration features.
“The free TerraGo Desktop turns Adobe Reader into a powerful geospatial application that gives users the ability to view, manipulate and update mapping data while leveraging Adobe collaboration capabilities to share information with others in the field and at home base.”
Is this a viable way of transporting cartographic intent?
How about Mapskins? “Skin” your map (with SLD?) the way you can “skin” a website with different stylesheets. Maybe I should register that as a trademark… [Edit:] uh oh – these guys beat me to it!
http://www.dataeast.ru/eng/4e_CarryMap.html
Pingback: James Fee GIS Blog » Blog Archive » GSS - Style your web maps with CSS client side
Pingback: Talking Spatially « Cecilia Wandiga's Blog
There is also the “Atlas Styler” open source to do just the editing of SLD: “The created SLD files are compatibel with GeoServer, uDig, Geotools-based applications and all other programs that use the OGC SLD standard.”.
http://en.geopublishing.org/AtlasStyler