SpatialTau v2.3 - GIS Agility

SpatialTau is my weekly newsletter that goes out every Wednesday. The archive shows up in my blog a month after the newsletter is published. If you’d like to subscribe, please do so here.

Work in the spatial field long enough and you’ll reinvent yourself over and over again. I’ve been cleaning up my old blog and it is amazing to me to see how much .NET/VB6/Oracle I used to do. Heck I used to be a big proponent of GeoDesign but not so much anymore. I remember the first GeoDesign Summit as a good time but the latest pictures from 2015 seem to show things have changed.

A lot of what we experience clearly affects how we approach our work as we move along in life. All that fighting ArcSDE has helped me approach PostGIS better. All that fighting the Esri WebADF has helped me work with Node.js better. All that expended capital on GeoDesign has taught me not to be involved with company sponsored community efforts. None of it is lost though, it all helps built the future as to what Spatial IT becomes.

The news that Google is shutting down Google Maps Engine definitely caught people’s attention. But Google Maps API continues on and working with maps doesn’t really change. All that capital spent working with Google Maps Engine can just be rolled into the Google cloud platform easily and off you go. Years ago such an announcement would have had people jumping off the cliff but it’s just how applications work these days.

Being a GIS developer (whatever that is) has been a crazy ride. Every year you learn new languages, new libraries, new server technologies. That’s why I feel like we’re so lucky to be working in this space. The past year has been Node.js and Angular.js while this year is shaping up to be React and Go. It’s that change that is exciting, fresh and keeps us all working hard. Let the good times roll!

January 22, 2015 geodesign Thoughts

SpatialTau is my weekly newsletter that goes out every Wednesday. The archive shows up in my blog a month after the newsletter is published. If you’d like to subscribe, please do so here.

If there is one thing that you can take away from GIS software is it’s love of the toolbar. Every function has a little button on the toolbar. One can turn on all the toolbars in ArcGIS Desktop and you get this nightmare.

ArcGIS Toolbars

But it isn’t just Esri, QGIS almost as bad (possibly uglier).

QGIS Toolbars

I get some things are best done by having a toolbar, editing vectors for one seems so logical. But most are just some tool library that some programmer links up to some obscure bitmap graphic that represents what we’re trying to do. I’ve been struggling to think of a better way though.

Options

ModelBuilder/FME Workbench: The logical method to performing GIS analysis is a flow diagram. The data flows through analysis like water through pipes. At the bottom is the outcome (hopefully clean and pure). Rather than highlighting a feature/layer, you perform the selection with SQL-type statements and apply logic rather than luck.
Scripting: Goes somewhat hand-in-hand with above. The visualization of the scripting is handled by ModelBuilder/FME Workbench allowing the GIS analyst to show others what they are doing. As much as I do love scripting and GIS, the visualization methods used by ModelBuilder/FME Workbench allow sharing of the model with other who might not see the workflow.
Wizards: Yea I hate wizards but in a way they work better than a toolbar. They are very limited in what they do but it limits the user to the “rails” of the wizard workflow ensuring that they complete the analysis correctly. Most of the time the toolbars call wizards to complete a function. This is the method preferred by button pushers.
Intellisense: This was the great hope I’ve had with GIS software. All the power of the command line but all the modern features of an IDE. Esri has prototyped this for Python and ArcPy but it is sort of a hack rather than the preferred method. I always felt I was at my best spatial analysis with ARC/INFO back in the day and I’m sure when Esri/QGIS release such a tool integrated into the application I’ll go back to using it.

I love the clean look of a blank canvas for creativity and unfortunately GIS software just clutters up that zen with crazy Windows XP logic. ArcGIS Professional brings the ribbon interface which hides much of these toolbars but it’s still confusing and illogical (seriously, those tabs are a crap shoot for finding a tool). There has been much innovation with mobile and web mapping. Hopefully we’ll see ArcGIS Professional and QGIS start to push the envelope with their interfaces. Just because we’ve been doing this way since ArcView 2.x does mean it is right.

January 14, 2015 arches desktop qgis Thoughts ui

SpatialTau is my weekly newsletter that goes out every Wednesday. The archive shows up in my blog a month after the newsletter is published. If you’d like to subscribe, please do so here.

If there is one thing that you can take away from GIS software is it’s love of the toolbar. Every function has a little button on the toolbar. One can turn on all the toolbars in ArcGIS Desktop and you get this nightmare.

ArcGIS Toolbars

But it isn’t just Esri, QGIS almost as bad (possibly uglier).

QGIS Toolbars

I get some things are best done by having a toolbar, editing vectors for one seems so logical. But most are just some tool library that some programmer links up to some obscure bitmap graphic that represents what we’re trying to do. I’ve been struggling to think of a better way though.

Options

ModelBuilder/FME Workbench: The logical method to performing GIS analysis is a flow diagram. The data flows through analysis like water through pipes. At the bottom is the outcome (hopefully clean and pure). Rather than highlighting a feature/layer, you perform the selection with SQL-type statements and apply logic rather than luck.
Scripting: Goes somewhat hand-in-hand with above. The visualization of the scripting is handled by ModelBuilder/FME Workbench allowing the GIS analyst to show others what they are doing. As much as I do love scripting and GIS, the visualization methods used by ModelBuilder/FME Workbench allow sharing of the model with other who might not see the workflow.
Wizards: Yea I hate wizards but in a way they work better than a toolbar. They are very limited in what they do but it limits the user to the “rails” of the wizard workflow ensuring that they complete the analysis correctly. Most of the time the toolbars call wizards to complete a function. This is the method preferred by button pushers.
Intellisense: This was the great hope I’ve had with GIS software. All the power of the command line but all the modern features of an IDE. Esri has prototyped this for Python and ArcPy but it is sort of a hack rather than the preferred method. I always felt I was at my best spatial analysis with ARC/INFO back in the day and I’m sure when Esri/QGIS release such a tool integrated into the application I’ll go back to using it.

I love the clean look of a blank canvas for creativity and unfortunately GIS software just clutters up that zen with crazy Windows XP logic. ArcGIS Professional brings the ribbon interface which hides much of these toolbars but it’s still confusing and illogical (seriously, those tabs are a crap shoot for finding a tool). There has been much innovation with mobile and web mapping. Hopefully we’ll see ArcGIS Professional and QGIS start to push the envelope with their interfaces. Just because we’ve been doing this way since ArcView 2.x does mean it is right.

January 14, 2015 arches desktop qgis Thoughts ui

SpatialTau v2.1 - Scripting

SpatialTau is my weekly newsletter that goes out every Wednesday. The archive shows up in my blog a month after the newsletter is published. If you’d like to subscribe, please do so here.

GIS and scripting go hand in hand. With its roots in command line mainframe tools in the 70s and 80s, GIS required scripting to perform any analysis/cartography. The hardest thing was that each GIS package required a different scripting language; AML, SML, Avenue, MapBasic, Magik, LISP, VB-Script, JavaScript, Python, Perl, TCL, etc. In the past 5 years though, everyone from Autodesk and Esri to open source uses Python. It’s at the point that you write Python scripts to perform GIS analysis (well unless you’re a wizard-based GIS professional or what I like to call “button pusher GIS”).

I could spend hours writing about how Python has changed GIS analysis but I think we’ve talked about that enough. No one needs to sell Python to GIS professionals, it is just assumed as a requirement. I doubt I’d hire any GIS person who couldn’t write Python scripts to perform analysis. We all use it to the point where we don’t even think about using it. That said, I think JavaScript could be more important for the future.

When I think GIS and Python, I think classic GIS Analysts doing their thing with large multi-monitor displays and huge loud desktops. I hate to stereotype people but generally Python is old school GIS analysis. The world has moved to a JavaScript focused environment and we’re starting to see some amazing things being done with JavaScript (D3). I don’t think one can replace Python with JavaScript just yet but the tempo of innovation is clearly with JavaScript. If you’re in an Esri centric universe, Python is still the weapon of choice, but outside of that silo, JavaScript is where people are doing amazing things. I used to say, “Just learn Python” but I think that needs to read, “Learn JavaScript and Python”.

That doesn’t mean there isn’t some really cool stuff going on with Esri. Just 2 months ago, Esri said they were supporting the SciPy Library for spatial analysis. This is a huge shift and one that I think everyone should be excited about. SciPy is a great analysis tool that you can use anywhere, just not Esri. You don’t need to worry about an Esri license being available for ArcPy, just use it wherever you wish. Now short term this is probably just a niche thing as only the bleeding edge ArcGIS Pro supports it, but the plans are to back port it to the classic ArcGIS Desktop enabling you to work with your existing projects. I can’t stress enough that people should learn how to use SciPy with Esri if only to broaden their marketability.

So all this cool cutting edge Python/JavaScript means we’ll be standardizing on one or two scripting languages? Possibly, but I was just looking at my text editor to see what scripts I’ve used/written in the past few months. For me I see mostly .pl and .rb as extensions. That’s right, I seem to write mostly Perl and Ruby scripts. But does that mean I’m not following my own recommendation above? Possibly, I hold on to Perl out of habit as I’ve been using it since the early 90s. But I don’t use it for GIS analysis, I can clearly see that Python is the choice for that. Same with Ruby, but I blame that on my stint at WeoGeo which was a Ruby/Rails shop. But what is my point here?

I think it is that workflows we use daily have tools that best help get them done. I write this newsletter in Markdown and then use Gruber’s canonical Perl library to convert it to HTML. I also use Perl to slurp data off websites that don’t allow downloads. I use Perl to manage files on my computer and push to Amazon S3. Here is the kicker, I could do all of this with Python but I haven’t. I think the key is always to use the best scripting language to get the job done. That still means Python and JavaScript with GIS. Ruby and Perl are best left to where it makes sense and for most people I can simply say it doesn’t make any sense to use it.

Now let me go ahead and run the Perl script to convert this newsletter into HTML and send it off. I wish everyone a super 2015 and be ready for some amazing new spatial tools coming out in the next few months!

January 7, 2015 scripting Thoughts