The End of GIS

So yea, link bait title, sue me.  But I felt like it needed to be there.  But before I go into what that all means, I’m going to continue blogging over on Medium but with a new focus. The RSS feed, email blasts, and Twitter account will cease to produce original content.

So why is there an end to anything?  I’ve been working toward this end for some time, the focus has been to move away from proprietary stacks and toward open source.  But there is a more significant theme to this.  Pivoting away from specialized software that is good at one thing, towards libraries to get things done.  Regardless I’m now working at a company that specializes in aggregating, analyzing and visualizing 3D data.  GIS has been useful at many things, but 3D was never one of it.

The start of Spatially Adjusted happened over the course of a family vacation to my wife at the time’s in-laws in rural Texas.  I can’t recall exactly what made me start, but it was the intersection of Esri and Open Source.  This was pre-OSGeo, things in my life were still ArcGIS and mostly ArcIMS.  There was a ton about me being excited about EDN when that first arrived and unboxing ArcGIS 9.1.  But I was getting into open source.  In fact, the first time I blogged about PostGIS, Sean Gillies was quick to put me in my place.  Because of course I was a big Esri supporter, and all he saw was someone complaining about the quality of the software.

My blog has a big story arc in it.  I go from “Esri blogger” to “Esri hater.”  Early on I used to get Esri passing me info to get the word out.  The reality of this was there was no Twitter or Facebook yet, so the only place people could be open was my comments on my blog posts.  But over the years I grew bitter about the software.  I grew tired of competing against Esri on contracts.  I became angry at software being half-baked and having to rewrite things every few years.  Look, there was a ton to like about the Esri Web ADF…  No wait, there wasn’t.  I’m sure people worked very hard on it, and they probably take it personally when I call it a POS but it was.  Engineers aren’t at fault for creating the Web ADF, Esri marketing is at fault for choosing to push it.

I honestly could write pages on why I dislike things about Esri but I won’t.  I’m honestly over it.  I look back at ArcObjects, MapObjects, Web ADF and the rest and I feel like it was a different person.  I cannot picture myself doing that work anymore, and that’s OK, we all grow up and grow into what we enjoy.  That’s the big picture through this journey, being open to change.  The “threat” of Google Earth, the “threat” of open source, the “threat” of the ELA.  All irrelevant in the end.  The most prominent part of our professional lives is our ability to handle change.  Don’t assume anything, just look for ways to improve your workflows, provide better service toward others and be proud of your career.

Throughout this journey, there have been a couple people who have affected great change in me.  Early on I can only think of two people; Howard Butler and Sean Gillies.  Both forced me to look at how I perceived open tools such as GDAL, UMN MapServer, and PostGIS.  Sean more than anyone called out my proprietary bullshit and while I didn’t agree with everything he said, it did open my eyes.  Later on, blogging brought me into contact with more developers.  People such as Bill Dollins, Dave Bouwman, and Brian Flood.  The work they were doing, even in the Esri ecosystem really helped me grow.  Even inside Esri, the creation of EDN and the DevSummit introduced me to Brian Goldin, Steve Pousty and Rob Elkins who basically made the first DevSummit my Woodstock.

I also can’t stress enough how many people I’ve met over the years because of this blog.  Not a conference goes by where someone introduces themselves to me and tells me they follow me.  That means a ton as personal networks is what drives us all.  It has been those who introduce me to the fantastic stuff they are working on that inspires my passion.  But that is why I think my story arc went from “Esri blogger” to the intersection of 3D BIM and GIS.

I really can’t think of anyone I’ve met over these years I don’t have a ton of respect for.  From Art Haddad pushing ArcGIS Server to be something more than a hacked together project to Jim Barry always making sure I could find the right documentation or developer help, I’ve always been lucky enough to find the right person to help out.  I really could go, but everyone should know what a great asset you are and still will be.

So what now for me?  At Cityzenith I’m focused on building the platform that the real estate and AEC industries can use to make a better world.  This blog has been on so many different platforms over the years.  Best I can recall the progress went; Blogger -> MovableType -> WordPress -> Octopress -> WordPress -> Github Pages -> WordPress and rather than port it over to yet another platform I think it has earned the right to relax.  Just like PlanetGS.com got to retire in dignity, so will Spatially Adjusted.

So follow me over at Medium where I’ll be talking about Elastic, Unity, Mapbox, Turf.js, Tippecanoe, Safe FME, 3D formats, AWS (including Lex, Lambda and Comprehend) and using Unity inside web browsers and mobile devices.  Should be a blast!

So I think I’ll just leave this here because it is how I feel.

GIS has been won!

25 years of Geospatial

I was just thinking about this on my plane flight back from Cityzenith HQ. Spring of 1993 I was first exposed to what I later learned was called GIS because some guy in Canada had the smarts to label what we were all learning to do. My exposure in college was mostly because pen and paper cartography I was learning didn’t work out for me, a computer geek since my Dad brought home a TI-99/4a and I discovered you could write applications.

Of course, I bought my stupid pen set for some considerable amount of money (at least to a college student back then) and tried to draw the Puget Sound with any degree of accuracy. Seriously though, if you’ve ever seen me draw even a circle or a square you can pretty much guess how this “puget sound” looked like (I recall it looked like an eggplant crossed with a maple leaf). I recall sitting in the computer lab for the statistics class and seeing that on the Macintosh computer a copy of Aldus Freehand was there. I fired that up, drew a damn good looking Puget Sound and submitted it to the teacher. So I didn’t even get an F, he gave me an incomplete which was to be expected back then. Computers were for spreadsheets and reports, not cartography.

`

But while this was going on, I had zero idea that this problem had already been solved by many people, including those Canadians I mentioned earlier. But it took two more years for me to be connected to the mid-decade US census and my internship at the City of Mesa, AZ planning department until I was exposed to Arc/INFO 5 and ArcView 2 on Unix to see how the world of points, polylines and polygons existed. But there I was creating maps in Freehand and throwing SPSS tables on top of them. GIS as we know it today, but I had zero idea what the heck I was doing. Originally being an Economics major I had the concept of table + pie chart/line graph down pat. I just replaced the chart with a hand-drawn map.

This week has been about sun angles, shadow analysis and time sliders in Unity, but it’s hard not to think back to a time where my struggle against drawing a map by hand introduced me to computer maps and eventually a career for basically 25 years. I can see an alternate universe where 1993 James sucked it up, worked hard to draw the water body by hand and became a city planner in some small midwestern city. Thank god that didn’t happen, and I was able to grow from Arc/INFO -> ArcView -> PC ArcInfo -> ArcInfo Workstation -> ArcGIS Desktop -> FME Desktop -> uDIG -> QGIS and then on to the multitude of open source libraries I use in my day to day workflows.

While that is a fun path to think of, I get excited to my next 25 years in spatial. Working with Unity, OSM, Elastic and AWS Lambda I can see how what we do has such a great exciting future ahead of us. The world loves what we do, and we’re lucky enough to be able to do it every day. Don’t let anyone make you feel bad for using software or products because it’s not “hipster” or “open.” If you’re pushing the bar forward, that’s good enough.

The Matrix of Spatial Data

I was thinking this morning about how much of my professional life has been about vector data. From the moment I started using Macromedia Freehand in college in the early 90s (before I had heard about GIS) to make maps to the 3D work, I’m doing with Unity and Cityzenith I’ve used vector data. I wasn’t genuinely introduced to raster data until I started using ArcInfo 5 at my first internship and working with grids and even then it was still about coverages and typing “build” or “clean” again and again. We did a bunch of raster analysis with Arc, but mostly it was done in Fortran by others (I never was able to pick up Fortran for some reason, probably best in the long run).

It’s easy to see and use vectors in professional spatial work for sure. I always feel like Neo from the Matrix, I look at features in the world and mentally classify them as vectors:

  • Bird -> point
  • Electrical transmission line -> line
  • House -> polygon

Heck or how you might think of a bird as a point (sighting), line (migratory pattern) or polygon (range). So damn nerdy and my wife fails to see the fun in any of this. Again, like Neo when he finally sees the world like the Matrix truly is we see things as the basic building blocks of vector data.

As I’m flying to Chicago this morning and I stare out the window of the airplane, I can’t help but think of rasters though. Sort of like that hybrid background we throw on maps, the world beneath me is full of opportunities to create vectors. Plus I bet we could run some robust agriculture analysis (assuming I even knew what that was) to boot. The world is not full of 1s and 0s but full of rasters and vectors.

As I’m a point, traveling a line on my way to a polygon, I can’t help but appreciate the spatial world that has been part of my life for over 20 years. I can’t help but think the next 20 is going to be amazing.

Plus Codes; Another Attempt at Addressing Places Without Street Addresses

Yes everyone knows about What3Words.  It was an attempt to come up with an easy way to assign addresses for places where there are none.  In the end, a proprietary addressing system will never gain traction, and of course the inevitable eventually happened. My personal feeling is that What3Words never really got us beyond x/y numbering and the logic behind an addressing system was not there.  Enter Plus codes which comes at this problem from a different perspective.  There is a very detailed analysis of existing methods and why they choose to go this direction that I’ll leave it up to you to read.

Probably the biggest reason to pay attention is that this open addressing system was developed by Google.  In fact, they are already implementing it in India as we speak which goes a very long way to making this happen.

All these systems are built on the idea the world is a grid, and how deeply you drill down into that grid is your address so things need not be a single point, they can be an area which opens up many exciting ideas for addressing, especially outside of North America and Europe.  Check out the Github project to learn more.

Focus on Data

When you think geospatial you think data, right? You imagine GIS professionals working their butts off making normalized datasets that have wonderful metadata. Nah, that’s just some slide at the Esri UC where “best practices” become the focus of a week away from the family in the Gaslamp. For some reason, GIS has become more about the how we do something and less about the why we do something. I guess that all that “hipster” and “technologist” thinking that goes into these “best practices” loses the focus on why we do what we do, the data.

At Cityzenith the first question a customer asks me is what data do we have available. See that’s because they aren’t GIS technologists, they’re just working folk who have to solve a problem. That problem requires the same problem that an accountant requires, accurate data. The last question these people care about is “Should I script this with JavaScript, Python or Ruby?”. They’re just looking for data that they can combine with their proprietary company data to make whatever decisions they need to make.

Finding Data is Hard

So much of what we do in our space is wasted on the tools to manage the data anymore. Sure in the 90s we needed to create these tools, or improve them so they could rely on enough to get our work done. But the analysis libraries are basically a commodity at this point. I can probably find 100 different ways to perform a spatial selection on GitHub to choose from. Personally, I can’t even recall opening ArcGIS or QGIS to solve a problem. There just isn’t a need to do so anymore. These tools have become so prevalent that we don’t need to fight battles over which one to use anymore.

Your TIGER WMS is available

Thanks to Google and OpenStreetMap, base maps are now commoditized to the point that we rarely pay for them. That part we can be sure that we’ve got the best data. (Disclosure, Cityzenith users Mapbox for our base mapping) But everything else is still lacking. I won’t pick on any vendor of data but generally, it works the same way, you either subscribe to a WMS/WFS feed (or worse, some wacky ArcGIS Online subscription) and if you’re “lucky”, a downloaded zip file of shapefiles. Neither lends itself to how data is managed or used in today’s companies.

Back to our customers, they expect a platform that can visualize data and one that is easy to use. But I know the first question they ask before signing up for our platform is, “What data do you have?”. They want to know more about our IoT data, data from our other partners (traffic, weather, demographics, etc.) and how they can combine it with their own data. They will ask about our tech stack from time to time, or how we create 3D worlds in the browser but that is so rare. It’s:

  1. What do you have?
  2. Where do you have it?

There are so many choices people have on how they can perform analysis on data. Pick and choose, it’s all personal preference. But access to the most up-to-date, normalized, indexed and available data for their area of interest. That’s why our focus has been partnering with data providers who have these datasets people need and present them to our users in formats and ways that are useful to them. Nobody wants a shapefile. Get over it. They want data feeds that they can bring into their workflows that have no GIS software in them whatsoever.

As I sit and watch the news from the Esri UC it is a stark reminder that the future of data isn’t in the hands of niche geospatial tools, it’s in the hands of everyone. That’s what we’re doing at Cityzenith.

Esri Arcade

When we think of Esri scripting and authoring languages, we think Python. Esri jumped in with two feet with Python and we were all much better off for it. But alas, as awesome as Python is, it isn’t as portable across the Esri ecosystem as they would like. To solve the problem either you choose another language to use that is more portable (JavaScript) or you write your own expression language and make is appear like Python and JavaScript had a baby.

At the Esri Arcade

Well that’s what Esri did, take Python, take JavaScript and a new expression language. Now in an open world this would be great because anything I wrote in Arcade would be usable anywhere else. But this is an Esri only solution as I can’t imagine other companies jumping in on it. But in Esriland, that’s OK because the ecosytem is large enough to support learning a proprietary language.

I don’t use Esri software anymore so I can’t play with it but it is a logical solution to their problem of having to write code to work with data in different platforms. Theoretically one can now use Arcade to author and render maps and let the Esri software handle the rest. I’d wait to see what happens with Arcade and the eventual 1.x release. It’s the Esri Web ADF talking but…

So another proprietary scripting language…

GIS Software has to be Hard to Use

Serious though, right? GIS has been defined by those who create much of it at “Scientific Software”. Because of such, it needs to be:

  1. Expensive
  2. Difficult to use
  3. Poorly documented
  4. Buggy
  5. Slow
ArcGIS Toolbars

Professional GIS*

GIS software is literally the kitchen sink. Most GIS software started out as a project for some company and then morphed into a product. They are a collection of tools created for specific projects duct taped together and sold as a subscription. We’ve talked about re-imagining how we work with spatial data but we rarely turn the page. The GIS Industrial Complex (open source and proprietary, everything is awful) is built upon making things hard to do. There has been attempts to solve the problem but then in themselves are usually built for a project rather than a product. Somewhat cynical but you have to wonder if this is true.

Tools such as Tableau are the future and as they add more spatial capability GIS Specialists will be out of a job. Being a button pusher seems more and more like a dead end job.