Spatially Adjusted

Category: Thoughts

  • SpatialTau v2.3 – GIS Agility

    SpatialTau is my weekly newsletter that goes out every Wednesday. The archive shows up in my blog a month after the newsletter is published. If you’d like to subscribe, please do so here.

    Work in the spatial field long enough and you’ll reinvent yourself over and over again.  I’ve been cleaning up my old blog and it is amazing to me to see how much .NET/VB6/Oracle I used to do.  Heck I used to be a big proponent of GeoDesign but not so much anymore.  I remember the first GeoDesign Summit as a good time but the latest pictures from 2015 seem to show things have changed.

    A lot of what we experience clearly affects how we approach our work as we move along in life.  All that fighting ArcSDE has helped me approach PostGIS better.  All that fighting the Esri WebADF has helped me work with Node.js better.  All that expended capital on GeoDesign has taught me not to be involved with company sponsored community efforts.  None of it is lost though, it all helps built the future as to what Spatial IT becomes.

    The news that Google is shutting down Google Maps Engine definitely caught people’s attention.  But Google Maps API continues on and working with maps doesn’t really change.  All that capital spent working with Google Maps Engine can just be rolled into the Google cloud platform easily and off you go.  Years ago such an announcement would have had people jumping off the cliff but it’s just how applications work these days.

    Being a GIS developer (whatever that is) has been a crazy ride.  Every year you learn new languages, new libraries, new server technologies.  That’s why I feel like we’re so lucky to be working in this space.  The past year has been Node.js and Angular.js while this year is shaping up to be React and Go.  It’s that change that is exciting, fresh and keeps us all working hard.  Let the good times roll!

  • SpatialTau v2.2 – The Toolbar

    SpatialTau is my weekly newsletter that goes out every Wednesday. The archive shows up in my blog a month after the newsletter is published. If you’d like to subscribe, please do so here.

    The Toolbar

    If there is one thing that you can take away from GIS software is it’s love of the toolbar. Every function has a little button on the toolbar. One can turn on all the toolbars in ArcGIS Desktop and you get this nightmare.

    ArcGIS Toolbars

    But it isn’t just Esri, QGIS almost as bad (possibly uglier).

    QGIS Toolbars

    I get some things are best done by having a toolbar, editing vectors for one seems so logical. But most are just some tool library that some programmer links up to some obscure bitmap graphic that represents what we’re trying to do. I’ve been struggling to think of a better way though.

    Options

    1. ModelBuilder/FME Workbench: The logical method to performing GIS analysis is a flow diagram. The data flows through analysis like water through pipes. At the bottom is the outcome (hopefully clean and pure). Rather than highlighting a feature/layer, you perform the selection with SQL-type statements and apply logic rather than luck.
    2. Scripting: Goes somewhat hand-in-hand with above. The visualization of the scripting is handled by ModelBuilder/FME Workbench allowing the GIS analyst to show others what they are doing. As much as I do love scripting and GIS, the visualization methods used by ModelBuilder/FME Workbench allow sharing of the model with other who might not see the workflow.
    3. Wizards: Yea I hate wizards but in a way they work better than a toolbar. They are very limited in what they do but it limits the user to the “rails” of the wizard workflow ensuring that they complete the analysis correctly. Most of the time the toolbars call wizards to complete a function. This is the method preferred by button pushers.
    4. Intellisense: This was the great hope I’ve had with GIS software. All the power of the command line but all the modern features of an IDE. Esri has prototyped this for Python and ArcPy but it is sort of a hack rather than the preferred method. I always felt I was at my best spatial analysis with ARC/INFO back in the day and I’m sure when Esri/QGIS release such a tool integrated into the application I’ll go back to using it.

    I love the clean look of a blank canvas for creativity and unfortunately GIS software just clutters up that zen with crazy Windows XP logic. ArcGIS Professional brings the ribbon interface which hides much of these toolbars but it’s still confusing and illogical (seriously, those tabs are a crap shoot for finding a tool). There has been much innovation with mobile and web mapping. Hopefully we’ll see ArcGIS Professional and QGIS start to push the envelope with their interfaces. Just because we’ve been doing this way since ArcView 2.x does mean it is right.

  • SpatialTau v2.1 – Scripting

    SpatialTau is my weekly newsletter that goes out every Wednesday. The archive shows up in my blog a month after the newsletter is published. If you’d like to subscribe, please do so here.

    GIS and scripting go hand in hand.  With its roots in command line mainframe tools in the 70s and 80s, GIS required scripting to perform any analysis/cartography.  The hardest thing was that each GIS package required a different scripting language; AML, SML, Avenue, MapBasic, Magik, LISP, VB-Script, JavaScript, Python, Perl, TCL, etc.  In the past 5 years though, everyone from Autodesk and Esri to open source uses Python.  It’s at the point that you write Python scripts to perform GIS analysis (well unless you’re a wizard-based GIS professional or what I like to call “button pusher GIS”).

    I could spend hours writing about how Python has changed GIS analysis but I think we’ve talked about that enough.  No one needs to sell Python to GIS professionals, it is just assumed as a requirement.  I doubt I’d hire any GIS person who couldn’t write Python scripts to perform analysis.  We all use it to the point where we don’t even think about using it.  That said, I think JavaScript could be more important for the future.

    When I think GIS and Python, I think classic GIS Analysts doing their thing with large multi-monitor displays and huge loud desktops.  I hate to stereotype people but generally Python is old school GIS analysis.  The world has moved to a JavaScript focused environment and we’re starting to see some amazing things being done with JavaScript (D3).  I don’t think one can replace Python with JavaScript just yet but the tempo of innovation is clearly with JavaScript.  If you’re in an Esri centric universe, Python is still the weapon of choice, but outside of that silo, JavaScript is where people are doing amazing things.  I used to say, “Just learn Python” but I think that needs to read, “Learn JavaScript and Python”.

    That doesn’t mean there isn’t some really cool stuff going on with Esri.  Just 2 months ago, Esri said they were supporting the SciPy Library for spatial analysis.  This is a huge shift and one that I think everyone should be excited about.  SciPy is a great analysis tool that you can use anywhere, just not Esri.  You don’t need to worry about an Esri license being available for ArcPy, just use it wherever you wish.  Now short term this is probably just a niche thing as only the bleeding edge ArcGIS Pro supports it, but the plans are to back port it to the classic ArcGIS Desktop enabling you to work with your existing projects.  I can’t stress enough that people should learn how to use SciPy with Esri if only to broaden their marketability.

    So all this cool cutting edge Python/JavaScript means we’ll be standardizing on one or two scripting languages?  Possibly, but I was just looking at my text editor to see what scripts I’ve used/written in the past few months.  For me I see mostly .pl and .rb as extensions.  That’s right, I seem to write mostly Perl and Ruby scripts.  But does that mean I’m not following my own recommendation above?  Possibly, I hold on to Perl out of habit as I’ve been using it since the early 90s.  But I don’t use it for GIS analysis, I can clearly see that Python is the choice for that.  Same with Ruby, but I blame that on my stint at WeoGeo which was a Ruby/Rails shop.  But what is my point here?

    I think it is that workflows we use daily have tools that best help get them done.  I write this newsletter in Markdown and then use Gruber’s canonical Perl library to convert it to HTML.  I also use Perl to slurp data off websites that don’t allow downloads.  I use Perl to manage files on my computer and push to Amazon S3.  Here is the kicker, I could do all of this with Python but I haven’t.  I think the key is always to use the best scripting language to get the job done.  That still means Python and JavaScript with GIS.  Ruby and Perl are best left to where it makes sense and for most people I can simply say it doesn’t make any sense to use it.

    Now let me go ahead and run the Perl script to convert this newsletter into HTML and send it off.  I wish everyone a super 2015 and be ready for some amazing new spatial tools coming out in the next few months!

  • SpatialTau v1.5 – Santa Bring Turf

    SpatialTau is my weekly newsletter that goes out every Wednesday. The archive shows up in my blog a month after the newsletter is published. If you’d like to subscribe, please do so here.

    I’m on holiday this week for obvious reasons but there is one bit of news that came out that has my attention.

    Turf: GIS for web maps

    Via Mapbox:

    Turf is GIS for web maps. It’s a fast, compact, and open-source JavaScript library that implements the most common geospatial operations: buffering, contouring, triangular irregular networks (TINs), and more. Turf speaks GeoJSON natively, easily connects to Leaflet, and is now available as a Mapbox.js plugin on our cloud platform.

    I love this, client side GIS in the browser.  I told someone a couple weeks ago that they should think about JavaScript as a GIS scripting engine.  Don’t toss the Python just yet but this is the clearest sign that we’re not far away from tossing the old GIS servers out on the street.

    Merry Christmas everyone!

  • SpatialTau v1.4 – Backend Irrelevance

    SpatialTau is my weekly newsletter that goes out every Wednesday. The archive shows up in my blog a month after the newsletter is published. If you’d like to subscribe, please do so here.

    What We See

    Do you care about the Google Maps backend?  I mean do you really think about how their server stack is run or managed?  Of course not, Google has successfully abstracted out the server part of Google Maps to the point we just assume it will always be available and running.  But it isn’t just Google, Esri and their ArcGIS Online (or whatever they call it) just runs.  Sure it has its idiosyncrasies that make us all angry and frustrated but as with Google we just assume it will always be available and running.  Back when I worked at WeoGeo, SLAs were very important to how we did business.  Our SLA and our data provider’s SLA were so important to how we did business.  Most people I talked to who wanted to build upon our platform were interested in what our SLA was.  I know Google and Esri have SLAs available but I rarely see people curious about what they are and build contracts around them.  They are assume it will always be there and always be available.

    To the Nines

    High availability is something that is always talked about with these services.  How many “nines” is your device available?  That was a point of pride.  Of course if you built upon Amazon or another provider, you could only offer as good as their SLA is.  One picked providers that have “high nines” in availability so that you could pass on that SLA to your own customers.  Heck, it was why you outsourced the hosting, even 3 nines is an incredible uptime level.  What about Esri’s SLA?  Well, I found this PDF dated from last year:

    Esri will use commercially reasonable efforts to make the Covered Services available with a Quarterly Uptime Percentage of ninety-nine point nine percent (99.9%) (“Service Commitment”).

    That’s “three nines” which is just about 9 hours downtime a year which is noot bad for most GIS applications.  But I wonder how many of us looked at it.  OK, what about Google Maps for Work SLA (Google renames this product way too much)?

    Google will use reasonable commercial efforts to provide Maps API web and mobile interfaces that are operating and available to Customers 99.9% of the time in any calendar month.

    Gee, pretty much exactly the same as Esri.  Now I have no idea what Esri’s or Google’s true availability has been over the course of the year.  I haven’t heard anyone complain about ArcGIS Online lately nor have I ever heard someone complain about Google Maps for Work.  One has to assume they are both running at least 99.9% availability.

    Should We Care?

    I think the answer is absolutely.  As we move more and more into hosting our services, we need to be more and more aware of what we are getting ourselves into.  I’d wager that 8-9 hours on average downtime a year is fine with most applications.  But that needs to be take into consideration when migrating your legacy self hosted GIS applications into “cloud-like” environments.  I know as a contractor, I’m always keeping my clients aware of what it means to be hosted in Google, Microsoft or Amazon’s cloud including the SLAs.  Eventually Google, Microsoft, Amazon, Rackspace, etc will have a downtime that will affect your applications.  Death, Taxes and a server crashing going down are the only guarantees in life.  We need to plan for this inevitability and have plans to alert our users/clients to this possibility.  Sticking your head in the sand and ignoring the problem is not going to help.  Being proactive and having a plan will.  Take a look at your hosted services, the SLA for each and what you can do to mitigate the failure or one or more.  You’ll be glad you did!

  • SpatialTau v1.3 – Just Like Google

    SpatialTau is my weekly newsletter that goes out every Wednesday. The archive shows up in my blog a month after the newsletter is published. If you’d like to subscribe, please do so here.

    The RFP

    I ran into this RFP a couple weeks ago that was a small business set-aside so URS couldn’t go after it.  I like to read most RFPs that come on to me desk though because they help me understand what customers are looking for.  This was a traditional “enterprise GIS” RFP where the client wanted someone to come in and clean up their geodatabases, migrate from SDE to Oracle Spatial and then create a new web front end (the old one was a classic Esri big button nightmare) that they wanted to be “just like Google”.  At the time I just let that fall out of my thoughts, but it stuck into my consciousness.  I mean, what does that even mean?

    Just like Google

    Now we all have an idea what it means to be Google, especially with mapping and data.  I can think of a slippy map with some points on it, a search bar at the top that magically finds the results you want.  Some pretty neat JavaScripting on the front end and overall responsiveness that makes you not even think about some clunky GIS server on the back end.

    Reality Bites

    The problem with that description is that what I just described was mostly front end client design and not the real power of Google.  There are plenty of great looking, very functional websites out there that are performant but not at the scale or responsiveness that Google is.  When someone writes they are looking for something “just like Google” they are also asking for the infrastructure that goes with it.  And that infrastructure rarely is “enterprise GIS”.  I can use the same tools that Google does for their applications but clearly their army of engineers is better than me.  I can leverage many of their tools but they don’t interact with enterprise solutions as deep at many RFPs require.  There is a spirit of “Google-like” that many try to deliver, but to actually deliver something “just like Google” is virtually impossible.

    Bad RFP or Bad Contractor

    I’ll be clear, I respond to many RFPs with “Google-like” features but I try and set expectations and constraints on what it can do.  But the expectation of “just like Google” is whose fault?  One could say the RFP writer doesn’t understand what the statement means.  But at the same time, it is contractors who love to throw out buzzwords such as “big data”, “cloud ready” and “responsive design” that ultimately should be blamed.  Software vendors routinely oversell their software and leave their users unhappy.  Contractors do this as well and it hurts long term relationships with their clients.

    Simple Always Wins

    I keep telling myself for each proposal I submit, simple always wins.  Simple isn’t claiming that something is “Google-like”.  Simple is spelling out what it means to be such a thing.  When we rely on buzzwords to describe what we do, or what our product does, or even our job titles; it obscures why we do what we do.  A simple road map as to what your solutions requires for the RFP goes miles beyond a confusing, copy and paste RFP that contradicts itself on every page.  For contractors, delivering a simple road map as to what your solution does helps just as much.  We all see great presentations and proposals every day.  They all have one thing in common.  Clear, concise recognition of the problem and then a solution that is easily understood and actionable.  I can only hope that I accomplish that every time.

  • SpatialTau v1.2 – Tilting at the Shapefile

    SpatialTau is my weekly newsletter that goes out every Wednesday. The archive shows up in my blog a month after the newsletter is published. If you’d like to subscribe, please do so here.

    Tilting at the Shapefile

    Now I’m sure if I went back to my blog and searched for how many times I’ve tried to kill off the shapefile even I would be surprised at how many times I’ve blogged about it.  Thus it seems about for the second newsletter I’ve ever written to focus on the “Shapefile Problem”

    The Problem

    So what exactly is this problem?  I mean what is so bad about a well supported, somewhat open file format?  I’ve told this story before but it never hurts to repeat.  My dad was borrowing my laptop a couple years ago and commented about all these DBF files all over my desktop.  He wondered why on earth would I have a format that he used in the late 80’s and outgrew because of it’s limitations.  Well I proceeded to explain to him the shapefile and how it worked and he just laughed.  That’s right, my 72 year old dad laughs at us wankers and our shapefile.  The DBF is only half the problem with the shapefile.  It doesn’t understand topology, only handles simple features (ever try and draw a curve in a shapefile?), puny 2GB file size limitation and not to mention you can’t combine points, polygons and lines in one file (hence every shapefile name has the word point, line or poly in it).

    Oh and it’s anywhere between 3 and 15ish file types/extensions.  Sure 3 are required but the rest just clutter up your folders.  I love the *.shp.xml one especially because clearly they thought so much about how to render metadata.  If I had a penny for every time someone emailed me just the *.shp file without the other two I’d be a rich man.  Heck just the other day I got the *.shp and *.dbf but not the *.shx.  Just typing the sentence makes me cringe.

    The Contenders

    1. The File Geodatabase (FGDB):  Esri’s default format for their tools.  It’a spatial database in a folder format.  The less mentioned about the Personal Geodatabase, the better.  But unlike most companies in the past 5 years, it isn’t built on SQLite, but Esri proprietary geodatabase format.  There isn’t anything inherently wrong with Esri taking this path but it means you’re stuck using their software or their APIs to access the file format.  To me this severely limits the FGDB to me an interchange file format and I think that is perfectly fine with Esri as they don’t really care too much if the FGDB doesn’t work with other’s software.  I’d link to an Esri page that describes the FGDB but there isn’t one. It’s a secret proprietary format that even Esri doesn’t want to tell you about.
    2. SpatiaLite: SpatiaLite has everything going for it.  It’s a spatial extension to SQLite which means at its core it’s open.  It’s OGC Simple Features compliant.  It is relatively well supported by GIS software (even Esri technically can support it with the help of Safe Software).  Plus it supports all those complex features that the shapefile can’t.  Heck OGC even chose it as the reference implementation for the GeoPackage (assuming people still care about that).  Heck supports rasters too!  But honestly, SpatiaLite was released in 2008 and hasn’t really made a dent into the market.  I can’t ever remember downloading or being sent a SpatiaLite file.  I’m guessing you can’t either.  I mean we all want a format that is similar to PostGIS and easily transferable (one file).  On paper that’s SpatiaLite.  But I think we have to chalk this up as Esri not supporting the format and it is relegated to niche use.
    3. GML/KML: Ron Lake probably loves I grouped these together but honestly they’re so similar in basic structure I’ve really just left them together.  My company uses KML quite a bit to share georeferenced photos.  That’s about it, pretty low use.  There is a ton of KML out there but it is mostly points.  There might be a ton of GML out there but I’m not Ron Lake.  KML is nice in the sense it has visualization included in the spec (you can make a line yellow) but it isn’t enough to get excited about.  It’s an OGC standard but as with SpatiaLite that doesn’t really seem to matter in the real world.  Don’t even try and use a different projection.  They have their use in specific cases but the limits of the formats means you’ll never see it being an interchange format.  Plus XML?  Oh and feel free to email me how GML is powerful because it supports OGC Simple features, I’ll still include it with KML.
    4. GeoJSON: It’s an open standard, so open in fact that OGC isn’t involved.  That’s a huge plus because mostly standards organizations do is make complex file formats for simple uses.  That’s not what GeoJSON is.  It can be many types of projections, it can be points, polygons and lines (with variations of many), it supports topology with the TopoJSON format and it’s JSON so it’s human readable.  But alas it isn’t supported by Esri so we run into the same problem as SpatiaLite.  BUT, Esri has shown interest in GeoJSON so there is hope that it will be well supported soon.  As with the shapefile/KML and unlike SpatiaLite it won’t support curves and other complex geometry or rasters and never will.  Thus it is not well suited as a shapefile replacement.
    5. Well Known Text (WKT): This comes out of the OGC and is used by software such as PostGIS for storage.  WKT supports lots of geometric objects (curves!) and TINs.  I’ve never been limited by WKT for vector files (you can almost feel where the end of this is going though) and many spatial databases from PostGIS and Oracle to SpatiaLite and SQL Server use the WKB (Well Known Binary) equivalent to store information.  But alas, we still don’t support rasters.  It’s a vector format for vector data.  SpatiaLite and the File Geodatabase both support rasters.

    There are many other formats but I think these are the only ones that really have any traction.  I could list formats such as GeoTIFF and say you could use that for rasters but you are limited to 4GB of data.  The vector guy in me wants to just say the heck with it all and use GeoJSON and WKT to solve the problem but given I’m still writing about this subject in December 2014 neither is a good solution.  We’re left with one simple truth…

    The Verdict

    The shapefile will outlive us all.  Unless Esri stops supporting it with their software at the same time as QGIS, Autodesk, etc it will continue to be the format that everyone uses.  In 2014 I’d wager 80% of all production geospatial data (I’m sandbagging here, probably this number is 95%) is stuck in the shapefile format where it resides comfortably.  Personally I’m a big fan of GeoJSON but I’ve started to get back into WKT lately and love the complex geometry support. If there is one thing I’ve learned in the past 20 years of “professional GIS” I’ve done, the shapefile is king.

  • SpatialTau v1.1 – Why a Newsletter?

    SpatialTau is my weekly newsletter that goes out every Wednesday. Archive shows up in my blog a month after the newsletter is published. If you’d like to subscribe, please do so here.

    Why a Newsletter?

    Earlier this month I turned off Planet Geospatial. It had been in operation for almost 10 years but honestly it peaked about 4 years ago and has been in a very slow decline. Blogs, while still critically important to our communicating with others, have taken a back seat to Twitter, Facebook and Tumblr. Heck, even I have made my blog dormant and moved my posting to Tumblr.

    But Tumblr has taught me one thing, a need for longer form writing. Tumblr, much like Twitter and Facebook is really meant for short quick thoughts that you want to get out fast. I originally thought I could move back to my old blog but the whole format seems limiting for me. Clearly what I really need is a format where I can write and get a bit deeper into my thoughts. Oddly enough the format I kept coming back to was a weekly newsletter. It’s a more relaxed format where I can take time to formulate my thoughts on a subject or subjects without that need to hit the publish button on a blog post.

    So this is SpatialTau, my weekly Spatial IT newsletter. It goes out every Wednesday and will be more in line with my older blog posts where I had more time to write and share my thoughts. I hope you enjoy it and share them with friends and colleagues.

    “What do you do?”

    Remember this question? I used to get it all the time and it was so hard to explain. I’d go into maps, databases and then the Internet. People sort of nod and seem to agree they understand just so you’ll stop talking about intersecting polygons and buffering the result. Then when Google Earth exploded on the scene, I’d used to just always say, “You know, like Google Earth…” and the other person would get all excited and say they looked up their hometown and saw their elementary school and how awesome it was that Google could find it.

    My fiancée’s mother asked me last week what I did. I started to go in with #opendata, #opengovernment (explaining hashtags along the way of course) and visualization. Unlike that Google Earth moment, lots of what we do is still very difficult for most people to really get their heads around. Sure they understand what it means to share data and make it open, but the process is still so difficult. I mean geospatial data is still locked up in that crazy File Geodatabase format which my fiancée’s mother would never begin to grasp. I was lucky enough to have some data I was working with in Google Drive so I showed her a spreadsheet view of it and she sort of got the idea. But going through the workflow of how I got it there is very foreign to everyone.

    I’m not pretending to say that spatial is special again, just that I think we’ve let the technology get ahead of the story. Even sharing a great blog post by the Sunlight Foundation about government data still gets that look that we used to get explaining an intersection of polygons. What really gets me is when you back into what we do from an open government perspective, allowing them to grasp the point of data being free and open, they start getting excited. But the tools we use are still very niche, very technical and very difficult to share. Rather than sharing how we do something, we need to be sharing why we do something. It’s the why that get’s peoples attention. It’s the reason why we do what we do that interests people. Then you can gage how much “what” they can take and decide if sharing the OpenLayers vs Leaflet.js debate is worth it. It’s hard for technologists to “break things down” because the excitement they feel is the touch and feel of the how we accomplish things. But the why is really the sexy part of our jobs.

    I really think we’re so lucky that Spatial IT has moved from the backrooms of GIS and into the front and center of the open data and open government movement. But we can’t lose sight that the world could care less about that great NPM module you wrote to massage spatial data. My soon to be mother-in-law gets the picture now and understands why what we do is so important. One person at a time.

    If you were lucky enough to be forwarded this newsletter from a friend, you can sign up here and get it delivered weekly to your inbox.

  • World Champs Again

    Of course, it is an even year. The Giants win again. See you in 2016!

  • San Francisco Giants to World Series – Again…

    It’s an even year so that means one thing, the San Francisco Giants win the World Series. Terrible we have to wait until Tuesday for Game 1 but it will be here soon.