The GIS Database

I’ve been thinking about GIS data a bit lately, mostly because I’m cleaning off old hard drives I’ve had in my possession to try and consolidate my data (or not lose the data off of old hard drives). Typically GIS data was accessed one of two ways, either from a server through some endpoint or via a local file store. I can’t look at these old ArcGIS Desktop MXDs anymore but I recall most of the work we did was local file store. You know, sitting on the “P drive” and referenced via a file path. We all remember opening up projects and seeing those red exclamation points telling us that data was moved (or the project file was).

It is very easy in retrospect to go back and call yourself batshit crazy for storing data this way (back up hopefully every night on a DLT tape). I mean think about this for a minute, nothing was versioned. We live in this world of git where everything I do (including this blog) is stored in a database where I can track changes and revert if need be. Now I’m not using this post to talk about the need of GeoGig or whatever that project is called these days (I’m not even sure it still exists), but the realization that GIS over the years is such a workgroup discipline.

I worked for AECOM, the largest AEC in the world. We did some amazing enterprise projects but GIS was never one of them. It was a small group of GIS “pros”, “doing” GIS to support some enterprise project that changed the world. Tacked on if you will, and it’s not just AECOM that worked that way. Every organization views GIS this way, like “graphics”. Why is this? Because GIS “pros” have let it be this way.

I’m not trying to come up with a solution here because I don’t think there is one. GIS is just very small minded compared to other professions in the tech space. Even the word “enterprise” has been appropriated to mean something totally different. Just having a web map does not make GIS “enterprise”, in fact all you’re doing is taking workgroup and making it worse. It is easy to pick on Esri (as I did above) but they’re not the big problem. It’s the implementations which make Esri have such terminology. That is, it is the GIS “pros” who cause these problems on themselves. Who is to fault Esri for trying to make a buck?

I have made it my professional career to fix broken GIS systems. People always ask me, “What madness you must see trying to undo broken GIS systems” but the reality is I see some amazing work. Just small minded implementations. It is easy to make fun of ArcObjects or GML but they are just libraries that people use to create tools.

This isn’t a call to arms or a reminder that you’re doing GIS wrong, it’s just thoughts on a plane headed across the country where I’m looking at data that I created as a workgroup project. I’m sure there are people cleaning up my work that I implemented in the past, I can tell you there is some bad choices in that work. Technology has caused many of us to lose being humble. And that results in only one thing, bad choices. In the end this is my reminder to be humble. The good thing is I have no shapefiles anywhere on this laptop. That’s a start.

Mr. Magoo Does GIS

I’m 46. It is weird even typing that. I’ll be 47 later this year which is even weirder. In my mind I think I’m till thirty-something but age is starting to creep up to me. I’ve noticed that I need reading glasses to see my iPhone.

Homer putting on reading glasses

The days of small text on small screens so I don’t have to scroll is over. Out is 11pt and 12pt fonts in my text editors and terminal windows and in comes 14pt. Fixed with fonts such as SF Mono and Roboto Mono seem to handle my eyes better too. Originally I was thinking that the dark mode on many terminal apps and text editors was going to be hard on my eyes but the fonts above on retina screens really pops for me. That said, dropping down to a non-retina monitor I have a very hard time reading things. So the quality of the screen and fonts seem to mean more to me than the color of the screen. Right now this is my environment:

For text editing, mostly I’m using BBEdit. I’ve hacked the SF Mono font so it is available for BBEdit to use and it is set at 14pt. For my theme, I’m using Xcode Dark which attempts to recreate the Xcode dark mode on BBEdit.

But I use VSCode as well. There I’m using Roboto Mono and the Dark+ Material theme. It is different than the look for BBEdit but it works for me in VSCode.

I’ve replaced Terminal.app with Hyper.is which I’m in love with. I use Roboto Mono and the Hyper-Clean theme. Again a little bit different than the above, but it just works. 14pt font as of course.

I think given that I have three different themes going on here is proof that I haven’t settled down on what looks best for me. I think eventually I’ll have a common theme color and go with it for all three products. 14pt font for me works well. It’s big enough that my eyes don’t strain, but small enough that I can fit enough on my screen. I think if Apple releases SF Mono as a system font, rather than a hack, I’ll go with it over Roboto Mono but honestly Roboto Mono is a great font for me too. We’ll just have to see what happens.

Most other apps I use on a regular basis such as Evernote, Safari, Chrome, Slack, I just go with the defaults. Many have a dark mode that mimics the Mac OS X dark mode and that’s fine with me. If Apple were to allow customization of that dark mode I’d probably be happy but we all know that will never happen. The last year has been hard for me with my eyes, it was the first time I felt myself holding my iPhone out at arms length to read it. What I’ve learned is to embrace a larger font and not strain my eyes. Pride is not suffering because you can’t read like you did 20 years ago, it’s having the will to make the choices you need to continue to be successful. For me, the above works.

Twitter Lists and Tweet Bankruptcy

With politics and hatred all over social media these days, it’s hard not to be nostagic of the Twitter we all enjoyed between 2008-2012. I look at my twitter feed these days and it isn’t focused. It’s probably just like yours, full of bots, yahoos, idiots and morons. I started looking through who I followed in the past few years and it’s not pretty. I really miss interacting with people on Twitter, rather than just posting memes.

So I thought about declaring tweet bankruptcy. Just delete the account and start fresh. But that’s not helpful. Sure I did it with my Facebook account but let’s be honest that’s just good practice. With Twitter though I don’t want to just blow away all my tweets (looks like over 41,000 of them), but reduce noise. Looking at my follows there are some basic groups:

  • Spatial Networks
  • Baseball
  • College Sports
  • Spatial/GIS
  • Programming
  • Humor
  • News

So then all I need to do is put everyone in lists (or multiple lists) and then I can segrate my twitter experience to my needs. I spent the weekend going through every follow I had (over 1,200) and move them into lists. But at the same time I culled my follows. I wanted to reduce it down to 200-400 follows. This way my main feed is what I consider value, but i can still enjoy conversations with people that aren’t follows.

It really has helped me get more value out of twitter. When I open Twitter on my phone I get only those accounts that I feel like are important enough to me that I should always see them. They all come out of those lists above. But then on my computer, I can use TweetDeck to have my lists always availabe and I can follow work related news or anything else with ease. The other nice thing is I can follow/unfollow people without worry that I’ll lose them. They will always be in my lists.

I can’t remember when Twitter had created lists but it has been a very long time. I resisted them because I thought the firehose was th best method for tweets but I’m enjoying this much more because I see tweets, especially in my Spatial/GIS list that I missed before because there was too much noise.

Friday Links

Back in the day I used to always have a Friday link blog post and I’ve noticed I’ve been doing a lot more reading so it just feels right to visit this back.

  1. Apple owes everyone an apology and it should start with me, specifically – You can’t but own one of the latest Apple MacBook Pros and not hate the keyboard. I’ve been “lucky” enough to experience all three versions of it. The latest is on my new laptop from Spatial Networks which I have to admit feels the best of any of them but I’m just waiting for the “f” key to stop working like it has on all my other ones. I used to enjoy typing on MacBooks but not anymore. The thing is they keep trying to fix broken and not just go back to something that worked.
  2. Electric scooters have zipped by docked bikes in popularity – Here in Tempe, AZ we get to see all of them. Bird, Uber/JUMP, Lime, Razor and various ones I can’t even tell the brand. Their are like lice on every corner just fallen over and broken. I noted in St. Pete that they didn’t have any scooters and it was surreal walking around on sidewalk without jumping out of the way of some idiot on a scooter. I don’t understand the business model but I hate to say they are here to stay.
  3. A look at IBM S/360 core memory: In the 1960s, 128 kilobytes weighed 610 pounds – I mean the title says it all. These things were HUGE! X and Y wires. It’s madness but apparently it worked!
  4. Notre Dame Cathedral will never be the same, but it can be rebuilt – Thanks to all the pictures and Lidar imagery, the Catheral will be rebuilt and be very close to original. But…

While architects have enough detailed information about the cathedral to pull off a technically very precise reconstruction, the craftsmanship is unlikely to be the same. Today, the stone that makes up the cathedral would be cut using machinery, not by hand by small armies of stonemasons as in the 12th century. “Nineteenth-century and 20th-century Gothic buildings always look a little dead, because the stone doesn’t bear the same marks of the mason’s hand,” Murray told Ars Technica.

Still I look forward to watching this happen.

The End of GIS

So yea, link bait title, sue me.  But I felt like it needed to be there.  But before I go into what that all means, I’m going to continue blogging over on Medium but with a new focus. The RSS feed, email blasts, and Twitter account will cease to produce original content.

So why is there an end to anything?  I’ve been working toward this end for some time, the focus has been to move away from proprietary stacks and toward open source.  But there is a more significant theme to this.  Pivoting away from specialized software that is good at one thing, towards libraries to get things done.  Regardless I’m now working at a company that specializes in aggregating, analyzing and visualizing 3D data.  GIS has been useful at many things, but 3D was never one of it.

The start of Spatially Adjusted happened over the course of a family vacation to my wife at the time’s in-laws in rural Texas.  I can’t recall exactly what made me start, but it was the intersection of Esri and Open Source.  This was pre-OSGeo, things in my life were still ArcGIS and mostly ArcIMS.  There was a ton about me being excited about EDN when that first arrived and unboxing ArcGIS 9.1.  But I was getting into open source.  In fact, the first time I blogged about PostGIS, Sean Gillies was quick to put me in my place.  Because of course I was a big Esri supporter, and all he saw was someone complaining about the quality of the software.

My blog has a big story arc in it.  I go from “Esri blogger” to “Esri hater.”  Early on I used to get Esri passing me info to get the word out.  The reality of this was there was no Twitter or Facebook yet, so the only place people could be open was my comments on my blog posts.  But over the years I grew bitter about the software.  I grew tired of competing against Esri on contracts.  I became angry at software being half-baked and having to rewrite things every few years.  Look, there was a ton to like about the Esri Web ADF…  No wait, there wasn’t.  I’m sure people worked very hard on it, and they probably take it personally when I call it a POS but it was.  Engineers aren’t at fault for creating the Web ADF, Esri marketing is at fault for choosing to push it.

I honestly could write pages on why I dislike things about Esri but I won’t.  I’m honestly over it.  I look back at ArcObjects, MapObjects, Web ADF and the rest and I feel like it was a different person.  I cannot picture myself doing that work anymore, and that’s OK, we all grow up and grow into what we enjoy.  That’s the big picture through this journey, being open to change.  The “threat” of Google Earth, the “threat” of open source, the “threat” of the ELA.  All irrelevant in the end.  The most prominent part of our professional lives is our ability to handle change.  Don’t assume anything, just look for ways to improve your workflows, provide better service toward others and be proud of your career.

Throughout this journey, there have been a couple people who have affected great change in me.  Early on I can only think of two people; Howard Butler and Sean Gillies.  Both forced me to look at how I perceived open tools such as GDAL, UMN MapServer, and PostGIS.  Sean more than anyone called out my proprietary bullshit and while I didn’t agree with everything he said, it did open my eyes.  Later on, blogging brought me into contact with more developers.  People such as Bill Dollins, Dave Bouwman, and Brian Flood.  The work they were doing, even in the Esri ecosystem really helped me grow.  Even inside Esri, the creation of EDN and the DevSummit introduced me to Brian Goldin, Steve Pousty and Rob Elkins who basically made the first DevSummit my Woodstock.

I also can’t stress enough how many people I’ve met over the years because of this blog.  Not a conference goes by where someone introduces themselves to me and tells me they follow me.  That means a ton as personal networks is what drives us all.  It has been those who introduce me to the fantastic stuff they are working on that inspires my passion.  But that is why I think my story arc went from “Esri blogger” to the intersection of 3D BIM and GIS.

I really can’t think of anyone I’ve met over these years I don’t have a ton of respect for.  From Art Haddad pushing ArcGIS Server to be something more than a hacked together project to Jim Barry always making sure I could find the right documentation or developer help, I’ve always been lucky enough to find the right person to help out.  I really could go, but everyone should know what a great asset you are and still will be.

So what now for me?  At Cityzenith I’m focused on building the platform that the real estate and AEC industries can use to make a better world.  This blog has been on so many different platforms over the years.  Best I can recall the progress went; Blogger -> MovableType -> WordPress -> Octopress -> WordPress -> Github Pages -> WordPress and rather than port it over to yet another platform I think it has earned the right to relax.  Just like PlanetGS.com got to retire in dignity, so will Spatially Adjusted.

So follow me over at Medium where I’ll be talking about Elastic, Unity, Mapbox, Turf.js, Tippecanoe, Safe FME, 3D formats, AWS (including Lex, Lambda and Comprehend) and using Unity inside web browsers and mobile devices.  Should be a blast!

So I think I’ll just leave this here because it is how I feel.

GIS has been won!

25 years of Geospatial

I was just thinking about this on my plane flight back from Cityzenith HQ. Spring of 1993 I was first exposed to what I later learned was called GIS because some guy in Canada had the smarts to label what we were all learning to do. My exposure in college was mostly because pen and paper cartography I was learning didn’t work out for me, a computer geek since my Dad brought home a TI-99/4a and I discovered you could write applications.

Of course, I bought my stupid pen set for some considerable amount of money (at least to a college student back then) and tried to draw the Puget Sound with any degree of accuracy. Seriously though, if you’ve ever seen me draw even a circle or a square you can pretty much guess how this “puget sound” looked like (I recall it looked like an eggplant crossed with a maple leaf). I recall sitting in the computer lab for the statistics class and seeing that on the Macintosh computer a copy of Aldus Freehand was there. I fired that up, drew a damn good looking Puget Sound and submitted it to the teacher. So I didn’t even get an F, he gave me an incomplete which was to be expected back then. Computers were for spreadsheets and reports, not cartography.

`

But while this was going on, I had zero idea that this problem had already been solved by many people, including those Canadians I mentioned earlier. But it took two more years for me to be connected to the mid-decade US census and my internship at the City of Mesa, AZ planning department until I was exposed to Arc/INFO 5 and ArcView 2 on Unix to see how the world of points, polylines and polygons existed. But there I was creating maps in Freehand and throwing SPSS tables on top of them. GIS as we know it today, but I had zero idea what the heck I was doing. Originally being an Economics major I had the concept of table + pie chart/line graph down pat. I just replaced the chart with a hand-drawn map.

This week has been about sun angles, shadow analysis and time sliders in Unity, but it’s hard not to think back to a time where my struggle against drawing a map by hand introduced me to computer maps and eventually a career for basically 25 years. I can see an alternate universe where 1993 James sucked it up, worked hard to draw the water body by hand and became a city planner in some small midwestern city. Thank god that didn’t happen, and I was able to grow from Arc/INFO -> ArcView -> PC ArcInfo -> ArcInfo Workstation -> ArcGIS Desktop -> FME Desktop -> uDIG -> QGIS and then on to the multitude of open source libraries I use in my day to day workflows.

While that is a fun path to think of, I get excited to my next 25 years in spatial. Working with Unity, OSM, Elastic and AWS Lambda I can see how what we do has such a great exciting future ahead of us. The world loves what we do, and we’re lucky enough to be able to do it every day. Don’t let anyone make you feel bad for using software or products because it’s not “hipster” or “open.” If you’re pushing the bar forward, that’s good enough.

The Matrix of Spatial Data

I was thinking this morning about how much of my professional life has been about vector data. From the moment I started using Macromedia Freehand in college in the early 90s (before I had heard about GIS) to make maps to the 3D work, I’m doing with Unity and Cityzenith I’ve used vector data. I wasn’t genuinely introduced to raster data until I started using ArcInfo 5 at my first internship and working with grids and even then it was still about coverages and typing “build” or “clean” again and again. We did a bunch of raster analysis with Arc, but mostly it was done in Fortran by others (I never was able to pick up Fortran for some reason, probably best in the long run).

It’s easy to see and use vectors in professional spatial work for sure. I always feel like Neo from the Matrix, I look at features in the world and mentally classify them as vectors:

  • Bird -> point
  • Electrical transmission line -> line
  • House -> polygon

Heck or how you might think of a bird as a point (sighting), line (migratory pattern) or polygon (range). So damn nerdy and my wife fails to see the fun in any of this. Again, like Neo when he finally sees the world like the Matrix truly is we see things as the basic building blocks of vector data.

As I’m flying to Chicago this morning and I stare out the window of the airplane, I can’t help but think of rasters though. Sort of like that hybrid background we throw on maps, the world beneath me is full of opportunities to create vectors. Plus I bet we could run some robust agriculture analysis (assuming I even knew what that was) to boot. The world is not full of 1s and 0s but full of rasters and vectors.

As I’m a point, traveling a line on my way to a polygon, I can’t help but appreciate the spatial world that has been part of my life for over 20 years. I can’t help but think the next 20 is going to be amazing.