So my last post was very positive. I figured out how to relate the teams that share a stadium with the stadium itself. This was important because I wanted to eliminate the redundant points that were on top of each other. For those who don’t recall, I have an example in this gist:
Now I mentioned that there were issues displaying this in GIS applications and was promptly told I was doing this incorrectly:
An array of <any data type> is not the same as a JSON object consisting of an array of JSON objects. If it would have been the first, I'd have pointed you (again) to QGIS and this widget trick https://t.co/4Wfy9OtOtM .
I had a conversation with Bill Dollins about it and he sums it up susinctly:
I get it, but “Do it this way because that’s what the software can handle” is an unsatisfying answer.
So I’m stuck, I honestly don’t care if QGIS can read the data, because it can. It just isn’t optimal. What I do care about is an organized dataset in GeoJSON. So my question that I can’t get a definitive answer, “is the array I have above valid GeoJSON code?”. From what I’ve seen, yes. But nobody wants to go on record as saying absolutely. I could say, hell with it I’m moving forward but I don’t want to go down a dead end road.
In a way it is good that Sean Gillies doesn’t follow me anymore. Because I can hear his voice in my head as I was trying to do something really stupid with the project. But Sheldon helps frame what I should be doing with what I was doing:
tables? what the? add , teams:[{name:"the name", otherprop: …}, {name:…}] to each item in the ballparks array and get that relational db BS out of your brain
Exactly! What the hell? Why was I trying to do something so stupid when the while point of this project is baseball ballparks in GeoJSON. Here is the problem in a nutshell and how I solved it. First off, let us simply the problem down to just one ballpark. Salt River Fields at Talking Stick is the Spring Training facility for both the Arizona Diamondbacks and the Colorado Rockies. Not only that, but there are Fall League and Rookie League teams playing there. Probably even more that I still haven’t researched. Anyway, GeoJSON Ballparks looks like this today when you just want to see that one stadium.
Let’s just say I backed myself in this corner by starting by only having MLB ballparks, none of which at the time of the project were shared between teams.
It’s a mess right? Overlapping points, so many opportunities to screw up names. So my old school thought was just create a one-to-many relationship between the GeoJSON points and some external table. Madness! Seriously, what was I thinking? Sheldon is right, I should be doing a JSON array for the teams. Look how much nicer it all looks when I do this!
Look how nice that all is? So easy to read and it keeps the focus on the ballparks.
The problem now is so many teams, especially in spring training, minor leagues and fall ball, share stadiums, that in GeoJSON-Ballparks, you end up with multiple dots on top of each other. No one-to-many relationship that should happen.”
The project had pivoted in a way I hadn’t anticipated back in 2014 and it was a sure a mess to maintain. So now I can focus on fixing the project with the Minor League Baseball realignment that went on this year and get an updated dataset in Github very soon.
One outcome of doing this nested array is that many GIS tools don’t understand how to display the data. Take a look at geojson.io:
geojson.io compresses the array into one big JSON-formatted string. QGIS and Github do this also. It’s an issue that I’m willing to live with. Bill Dollins shared the GeoJSON spec with me to prove the way I’m doing is correct:
3.2. Feature Object
A Feature object represents a spatially bounded thing. Every Feature
object is a GeoJSON object no matter where it occurs in a GeoJSON
text.
o A Feature object has a "type" member with the value
"Feature".
o A Feature object has a member with the name
"geometry". The value of the geometry member SHALL
be either a Geometry object as defined above or, in
the case that the Feature is unlocated, a JSON
null value.
o A Feature object has a member with the name
"properties". The value of the properties member is
an object (any JSON object or a JSON null value).
ANY JSON OBJECT! So formatting the files this way is correct and the way it should be done. I’m going to push forward on cleaning up GeoJSON Ballparks and let the GIS tools try and catch up.
Major League Baseball announced on Friday (February 12, 2021) a new plan for affiliated baseball, with 120 Minor League clubs officially agreeing to join the new Professional Development League (PDL). A full list of Major League teams and their new affiliates, one for each level of full-season ball, along with a complex league (Gulf Coast and Arizona) team, can be found below.
Minor League Baseball
What does that mean? Well for GeoJSON Ballparks basically every minor league team is having a modification to it. At a minimum, the old minor league names have changed. Take the Pacific Coast League that existed for over 118 years is now part of Triple-A West which couldn’t be a more boring name. All up and down the minor leagues, the names now just reflect the level of minor league the teams are. And some teams have moved from AAA to Single A and all around.
I usually wait until Spring Training is just about over to update the minor league teams but this year it almost makes zero sense. I’ve sort of backed myself into a spatial problem, unintended when I started. Basically, the project initially was just MLB teams and their ballparks. The key to that is that the teams drove the dataset, not the ballparks even though the title of the project clearly said it was. As long as nobody shared a ballpark, this worked out great. The problem now is so many teams, especially in spring training, minor leagues and fall ball, share stadiums, that in GeoJSON-Ballparks, you end up with multiple dots on top of each other. No one-to-many relationship that should happen.
So, I’m going to use this minor league realignment to fix what I should have fixed years ago. There will be two files in this dataset moving forward. One GeoJSON file of the locations of a ballpark and then a CSV (or other format) file containing the teams. Then we’ll just do the old fashioned relate between the two and the world is better again.
I’m going to fork GeoJSON-Ballparks into a new project and right the wrongs I have done against good spatial data management. I’m finally ready to play centerfield!
Now before we get too far, Apple has not created anything close to a Digital Twin as we know them. But what they have done is created an easy way to import your building models into Apple Maps. Apple calls this their Indoor Maps program.
Easily create detailed maps of your indoor spaces and let visitors see where they are right in your app. Organizations with large public and private spaces like airports, shopping centers, arenas, hospitals, universities, and private office buildings can register for the Indoor Maps Program. Indoor maps are built using industry standard tools and require only your existing Wi-Fi network to enable GPS-level location accuracy so visitors can navigate your spaces with ease.
Victoria Airport in the Apple IMDF Sandbox
OK, so clearly this is all about navigation. How do I know where I am in a building and how do I get to a place I need to be. Of course, this is somewhat interesting on your iPhone or iPad in Apple Maps, but clearly, there is more to this than just how do I find the restroom on floor 10 of the bank tower.
To load your buildings in Apple you need to use Mapkit or Mapkit.js and convert your buildings into Indoor Mapping Data Format (IMDF). IMDF is actually a great choice because it is GeoJSON and working toward being an OGC standard (for whatever that is worth these days). I did find it interesting that Apple highlights the following as the use case for IMDF:
Indoor wayfinding
Indoor routing
Temporal constraints
Connectivity amongst mapped objects
Location-based services
Query and find by location functionality
If you’re familiar with GeoJSON, IMDF will look logical to you:
I encourage you to review the IMDF docs to learn more but we’re talking JSON here so it’s exactly how you’d expect it to work.
Because IMDF buildings are generalized representations of the real-world data, this isn’t actually a Digital Twin. It also means that you need to do some things to your files before converting them to IMDF. Autodesk, Esri, and Safe Software all support IMDF so you should be able to use their tools to handle the conversions. I’ve done the conversion with Safe FME and it works very well and probably the best way to handle this. In fact, Safe has an IMDF validator which does come in handy for sure.
Safe FME support of IMDF
What does make moving your buildings to Apple’s Indoor platform is the new iPhone 12 and iPad Pro LiDAR support. This brings out some really great AR capabilities that become enabled with Apple’s devices. As I said last week, the LiDAR support in the current devices is more about getting experience with LiDAR use cases than actual LiDAR use. This is all about eventual Apple Glass (and Google Glass too) support and the AR navigation that can be done when you have hyper-accurate indoor models in your mapping software.
I’ve been dusting off my MapKit skills because I think not only is this capability useful for many companies but it really isn’t that hard to enable. I am spending some time thinking about how to use the extension capability of IMDF to see how IoT and other services can be brought in. Given the generalized nature of IMDF, it could be a great way to allow visualizing IoT and other services without the features of a building getting in the way. Stay tuned!
Taking this break I’ve been looking over my spatial data and trying to figure out how to best organize it. The largest public project I manage is the GeoJSON Ballparks and this one is easy to manage as it is just a Git repository with text files. GeoJSON makes sense here because it is a very simple dataset (x/y) and it has been used for mapping projects mostly which makes the GeoJSON format perfect. I used to maintain a Shapefile version of it in that repository but nobody ever downloaded it so I just killed it eventually.
But my other data projects, things I’ve mapped or worked on the past are in a couple of formats:
VECTOR
Shapefile
File Geodatabase
Personal Geodatabase
GeoJSON
KML
SpatiaLite
Raster
TIFF (mostly GeoTIFF)
Esri Grid
Now you can tell from some of these formats, I haven’t touched these datasets in a long time. Being Mac centric, the Personal Geodatabase is dead to me and given the modification dates on that stuff is 2005-2007 I doubt I’ll need it anytime soon. But it does bring of the question of archival, clearly PGDB isn’t the best format for this and I probably should convert it soon to some other format. Bill Dollins would tell me GeoPackage would be the best as Shapefile would cause me to lose data given limits of DBF, but I’m not a big fan of the format mostly because I’ve never needed to use it. Moving the data to GeoJSON would be good because who doesn’t like text formats, but GeoJSON doesn’t handle curves and while it might be fine for the Personal Geodatabase data, it doesn’t make a ton of sense for more complex data.
This is as close to a shapefile icon as I could find, tells you everything doesn’t it?
I’ve thought about WKT as an archival format (specifically WKB) which might make sense for me given the great WKT/WKB support in databases. But again, could I be just making my life harder than it needs to be just to not use the GeoPackage? But there is something about WKT/WKB that makes me comfortable for storing data for a long time given the long term support of the standard among so many of those databases. The practical method might be everything in GeoJSON except curves and those can get into WKT/WKB.
Raster is much easier given most of that data is in two fairly open formats. GeoTIFF or TIFF probably will be around longer than you or I and Esri grid formats have been well support through the years making both fairly safe. What are some limits to data formats that I do worry about?
File size, do they have limits to how large they can be (e.g. TIFF and 32-bit limit)
File structure, do they have limits to what can be stored (e.g. GeoJSON and curves)
File format issues (e.g. everything about the Shapefile and dbf)
OS centric formats (PGDB working only on Windows)
I think the two biggest fears of mine are the last two, because the first to can be mitigated fairly easily. My plan is the following; convert all vector data into GeoJSON, except where curves are required, I’m punting curves right now because I only have 3 datasets that require them and I’ll leave them in their native formats for now. The raster data is fine, TIFF and grid is perfect and I won’t be touching them at all. The other thing I’m doing is documenting the projects and data so that future James (or whomever gets this hard drive eventually) knows what the data is and how it was used. So little of what I have has any documentation, at least I’m lucky enough the file names make sense and the PDFs help me understand what the layers are used for.
One thing I’ve ignored through this, what to do with those MXDs that I cannot open at all? While I do have PDF versions of those MXDs, I have no tool to open them on Mac and even if I could, the pathing is probably a mess anyway. It bring up the point that the hardest thing to archive is cartography, especially if it is locked in a binary file like an MXD. At least in that case, it isn’t too hard to find someone with a license of ArcMap to help me out. But boy, it would be nice to have a good cartography archival format that isn’t some CSS thing.
I’ve been working at cleaning up all the GeoJSON-Ballparks records this past month. While the MLB stadiums and many of the AAA minor league teams have been updated, the international and small market teams have not. Some were out of data by almost 5 years. Long tail baseball stadiums are what they are and I’m working on automating much of this moving forward. The last two leagues that I’m updating are the Italian Baseball League and the German Bundesliga League. I hope to finish Germany tonight but I did post Italy yesterday.
Ballparks of the Italian Baseball League in GeoJSON
While Italy can’t go out and enjoy baseball just yet, at least their top tier baseball league has been mapped. If you’re looking for some live baseball, check out streaming Korean KBO League league (I’ve been watching the Giants of course). The next live stream will be on March 25th at approximately 7:40pm PDT.
GeoJSON-Ballparks is my favorite data project I’ve been part of. Probably because not only is it the best sport ever, but it is great keeping track of all the changes at ballparks through the years. MLB teams have mostly stopped building new ball parks so the changes are generally just updates to their names. This year the only new name was Truist Park. Oakland Coliseum reverted back from RingCentral which it never was able to become because of shenanigans. We do bring on a new ballpark in Arlington which is named almost the same as the old ballpark (Globe Life Field vs the old Globe Life Park in Arlington). Apparently the old stadium has been renovated to XFL standards so we should probably not call it a ballpark anymore. I just removed the old one since it is no longer a baseball stadium. I did the same thing with Turner Field.
I plan to review all the Spring Training Facilities of the Cactus League and the Grapefruit League and then review the AAA stadiums. We’ll have to see what happens with the MLB/MiLB negotiations. While it doesn’t affect the actual stadium points (at least in the short term, some of the fields could go away because of lack of support), the alignment of teams in leagues could be changed. So stay tuned and if you want to help out with the AAA stadiums, just create a pull request, would be greatly appreciated!
Oddly enough the biggest news this week from the iPhone 11 introduction by Apple barely got any play. In fact, on the iPhone 11 Pro website, you have to scroll past Dog Portrait mode to get any information about it. Apple describes the U1 chip thusly:
The new Apple‑designed U1 chip uses Ultra Wideband technology for spatial awareness — allowing iPhone 11 Pro to understand its precise location relative to other nearby U1‑equipped Apple devices.4 It’s like adding another sense to iPhone, and it’s going to lead to amazing new capabilities. With U1 and iOS 13, you can point your iPhone toward someone else’s, and AirDrop will prioritize that device so you can share files faster.4 And that’s just the beginning.
Makes sense right? A better way to AirDrop. But there is so much more there, “precise location relative to other nearby equipped Apple Devices“. But what is UWB and why does it matter? The UWB Alliancesays:
UWB is a unique radio technology that can use extremely low energy levels for short-range, high-bandwidth communications over a large portion of the radio spectrum. Devices powered by a coin cell can operate for a period of years without recharge or replacement. UWB technology enables a broad range of applications, from real-time locating and tracking, to sensing and radar, to secure wireless access, and short message communication. The flexibility, precision and low-power characteristics of UWB give it a unique set of capabilities unlike any other wireless technology.
So that’s really interesting, low energy use, high bandwidth and is very secure. I thought Jason Snell did a great job looking into the U1 on Six Colors:
From raw data alone, UWB devices can detect locations within 10 centimeters (4 inches), but depending on implementation that accuracy can be lowered to as much as 5 millimeters, according to Mickael Viot, VP of marketing at UWB chipmaker Decawave.
That’s pretty amazing. Basically it takes what makes Bluetooth LE great for discover, secures it and then makes it faster and more accurate. So we can see the consumer use cases for UWB, sharing files and finding those tiles we’ve heard so much about. But where this gets very interesting for our space is for data collection and working inside digital twins. You can already see the augmented reality use case here. A sensor has gone bad in a building, I can find it now with millimeter accuracy. But it’s not just what direction it’s how far. UWB uses “time of flight” to pinpoint location (measuring the time of signal to gauge distance), enabling it to know how far away it is. Just knowing a sensor is ahead of you is one thing, but knowing it is 20 feet away, that’s really a game changer.
You can see this through a little known app Apple makes called Indoor Survey. Small side note, back in late 2015 I blogged about Apple’s Indoor Positioning App which ties into all this. Where you really see this use is when you go to the signup page see how data is brought into this app using a standard called Indoor Mapping Data Format. Indoor Mapping Data Format (IMDF) provides a generalized, yet comprehensive data model for any indoor location, creating a basis for orientation, navigation and discovery. IMDF is output as an archive of GeoJSON files. Going to the IMDF Sandbox really shows you what this format is about.
Apple’s IMDF Sandbox
Basically you see a map editor that allows you to really get into how interiors are mapped and used. So Apple iPhone 11 UWB devices can help place themselves more accurately on maps and route users around building interiors. Smart buildings get smarter by the devices talking to each other. Oh and IMDF, Apple says, “For GIS and BIM specialists, there is support for IMDF in many of your favorite tools.“. I will need to spend a bit more time with IMDF but its basically GeoJSON objects so we already know how to use it.
The thing about GPS data collection is it works great outdoors, but inside it is much harder to get accuracy, especially when you need it. With Indoor Survey, devices can collect data much more accurately indoors because they know exactly where they are. If you’ve ever used Apple Maps in an airport and seen how it routes you from gate to gate, you get an idea how this works. But with UWB, you go from foot accuracy to sub centimeter. That’s a big difference.
Now we’re a long way away from UWB being ubiquitous like Bluetooth LE is. Right now as far as I can tell, only Apple has UWB chips in their devices and we don’t know how compatible this all is yet. But you can see how the roadmap is laid out here. UWB, GeoJSON and an iPhone 11. Devices help each other get better location and in turn make working with Digital Twins and data collection so much easier.
No curves. It’s a good point. GeoJSON and TopoJSON don’t support curves. But neither does Shapefiles. All three formats are meant to handle simple features. Points, polygons and line. Whereas TopoJSON handles topology, it still can’t draw true curves. But what’s the implication here? To share data that requires curves (it’s an edge case but still an important one) you have to use a proprietary format? Enter WKT. Well-known text supports much more vector types than the previous including curves. Following up on sharing data in common file formats, WKT fits the bill perfectly. Share your data as GeoJSON/TopoJSON, KML and Shapefile if needed, then use WKT for complex features. Still open completely and it is well supported with most open and proprietary software packages.
Sometimes you need to use curves and generally it does work out.
You may or may not have seen, but there is a Cuban/Tampa Bay Rays game going on today. Given the love of baseball between the two countries I’m sure we’ll see much more Cuban baseball over the next couple years. It just so happens that the GeoJSON-Ballparks project has all the professional baseball stadiums in Cuba already mapped in GeoJSON format, including Estadio Latinoamericano where the game is being played today. Enjoy!
