Apple’s Digital Twin is All About Augmented Reality
Now before we get too far, Apple has not created anything close to a Digital Twin as we know them. But what they have done is created an easy way to import your building models into Apple Maps. Apple calls this their Indoor Maps program.
Easily create detailed maps of your indoor spaces and let visitors see where they are right in your app. Organizations with large public and private spaces like airports, shopping centers, arenas, hospitals, universities, and private office buildings can register for the Indoor Maps Program. Indoor maps are built using industry standard tools and require only your existing Wi-Fi network to enable GPS-level location accuracy so visitors can navigate your spaces with ease.
Victoria Airport in the Apple IMDF Sandbox
OK, so clearly this is all about navigation. How do I know where I am in a building and how do I get to a place I need to be. Of course, this is somewhat interesting on your iPhone or iPad in Apple Maps, but clearly, there is more to this than just how do I find the restroom on floor 10 of the bank tower.
To load your buildings in Apple you need to use Mapkit or Mapkit.js and convert your buildings into Indoor Mapping Data Format (IMDF). IMDF is actually a great choice because it is GeoJSON and working toward being an OGC standard (for whatever that is worth these days). I did find it interesting that Apple highlights the following as the use case for IMDF:
- Indoor wayfinding
- Indoor routing
- Temporal constraints
- Connectivity amongst mapped objects
- Location-based services
- Query and find by location functionality
If you’re familiar with GeoJSON, IMDF will look logical to you:
{
"id": "11111111-1111-1111-1111-111111111111",
"type": "Feature",
"feature_type": "building",
"geometry": null,
"properties": {
"category": "parking",
"restriction": "employeesonly",
"name": {
"en": "Parking Garage 1"
},
"alt_name": null,
"display_point": {
"type": "Point",
"coordinates": [1.0, 2.0]
},
"address_id": "2222222-2222-2222-2222-222222222222"
}
}I encourage you to review the IMDF docs to learn more but we’re talking JSON here so it’s exactly how you’d expect it to work.
Because IMDF buildings are generalized representations of the real-world data, this isn’t actually a Digital Twin. It also means that you need to do some things to your files before converting them to IMDF. Autodesk, Esri, and Safe Software all support IMDF so you should be able to use their tools to handle the conversions. I’ve done the conversion with Safe FME and it works very well and probably the best way to handle this. In fact, Safe has an IMDF validator which does come in handy for sure.
Safe FME support of IMDF
What does make moving your buildings to Apple’s Indoor platform is the new iPhone 12 and iPad Pro LiDAR support. This brings out some really great AR capabilities that become enabled with Apple’s devices. As I said last week, the LiDAR support in the current devices is more about getting experience with LiDAR use cases than actual LiDAR use. This is all about eventual Apple Glass (and Google Glass too) support and the AR navigation that can be done when you have hyper-accurate indoor models in your mapping software.
I’ve been dusting off my MapKit skills because I think not only is this capability useful for many companies but it really isn’t that hard to enable. I am spending some time thinking about how to use the extension capability of IMDF to see how IoT and other services can be brought in. Given the generalized nature of IMDF, it could be a great way to allow visualizing IoT and other services without the features of a building getting in the way. Stay tuned!
November 4, 2020 apple Apple Glass augmented reality autodesk digital twin esri geojson Google Glass IMDF MapKit MapKit.js ogc safe Thoughts
COVID-19 is Showing How Smart Cities Protect Citizens
I feel like there is a before COVID and an after COVID with citizens’ feelings for Smart City technology. Now there is an election tomorrow in the United States that will probably dictate how this all moves forward and after 2016, I’ve learned to not predict anything when it comes to the current president. But, outside that huge elephant in the background, Smart City concepts have been thrust into the spotlight.
Photo by Michael Walter on Unsplash
Most cities have sent their non-essential workers home, so IoT and other feeds to their work dashboards have become critical to their success. The data collection and analysis of the pulse of a city is now so important that traditional field collection tools have become outdated.
Even how cities engage with their citizens has changed. Before COVID, here in Scottsdale, you needed to head to a library to get a library card in person. But since COVID restrictions, the city has allowed library card applications in person which is a huge change. The core structure of city digital infrastructure has to change to manage this new need. Not only engaging citizens deeper with technology but need to ensure those who don’t have access to the internet or even a computer are represented. I’ve seen much better smartphone access on websites over the summer and this will continue.
Even moving from a public space to a digital space for city council meetings has implications. The physicality of citizens before their elected leaders is a check on their power, but being a small zoom box in a monitor of zoom boxes puts citizens in a corner. Much will have to be developed to have a way for those who don’t wish to be in person be represented as well as those who choose to attend meetings in person.
COVID has also broken down barriers to sharing data. The imagined dashboard where Police, Fire, Parks & Rec, City Council, and other stakeholders have come to fruition. The single pane of glass where decision-makers can get together to run the city remotely is only going to improve now that the value has been shown.
Lastly, ignoring the possible election tomorrow, contact tracing, and other methods of monitoring citizens as they go around the city has changed mostly how people feel. Before COVID, the idea that a city could track them even anonymously scared the daylights out of people. But today we are starting to see the value in anonymous tracking so that not only we see who has been in contact with each other but how they interact in a city with social distancing restrictions.
Future planning of cities is changing and accelerated because of COVID. The outcome of this pandemic will result in cities that are more resilient, better managed, planned for social distancing, and are working toward carbon neutral environments. In the despair of this unprecedented pandemic, we see humanity coming together to create a better future for our cities and our planet.
The iPhone 12 Pro LiDAR Scanner is the Gateway to AR, But Not in the Way You Think
I’m sure everyone knows about it by now, the iPhone 12 Pro has a LiDAR scanner. Apple touts it to help you take better pictures in low light and do some rudimentary AR on the iPhone. But, what this scanner does today isn’t where the power will be tomorrow.
Apple cares a ton about photo quality, so a LiDAR scanner helps immensely with taking these pictures. If there is one reason today to have that scanner, it is for pictures. But the real power of the scanner is for AR. And AR isn’t ready today, no matter how many demos you see in Apple’s event. Holding up an iPhone and seeing how big a couch in your room is interesting, just as interesting as using your phone to find the nearest Starbucks.
Apple has spent a lot of time working on interior spaces in Apple Maps. They’ve also spent a ton of time working on sensors in the phone for positioning inside buildings. This is all building to an AR navigation space inside public buildings and private buildings in which owners share their 3D plans. But what if hundreds of millions of mobile devices could create these 3D worlds automatically as they go about their business helping users find that Starbucks?
The future is so bright though with this scanner. It helps Apple and developers get familiar with what LiDAR can do for AR applications. This is critically important on the hardware side because Apple Glass, no matter how little is known about it, is the future for AR. Same with Google Glass too, the eventual consumer product (ignoring the junk that the first Google Glass was) of these wearable AR devices will change the world, not so much in that you’ll see an arrow as you navigate to the Starbucks, but give you the insight into smart buildings and all the IoT devices that are around.
The inevitable outcome is in the maintenance of smart buildings
Digital Twins are valuable when they link data feeds to a 3D world that can be interrogated. But the real value comes when those 3D worlds can be leveraged using Augmented Reality to give owners, maintenance workers, planners, engineers, and tenants the information they need to service their buildings and improve the quality of building maintenance. The best built LEED building is only as good as the ongoing maintenance put on it.
The iPhone 12 Pro and the iPad Pro that Apple has released this year both have LiDAR to improve their use with photo taking and rudimentary AR, but the experience gained seeing the real-world use of consumer LiDAR in millions of devices will bring great strides to making these Apple/Google Glass devices truly usable in real-world use. I’m still waiting to get my iPhone 12, but my wife’s arrived today. I’m looking forward to seeing what the LiDAR can do.
October 29, 2020 apple augmented reality digital twin google iot iphone lidar Thoughts
Google AI Project Recreating Historical Streetscapes in 3D
When this caught my eye I got really interested. Google AI is launching a website titled re which reconstructs cities from historical maps and photos. You might have seen the underlying tool last month but this productizes it a bit. What I find compelling about this effort is the output is a 3D city that you can navigate and review by going in back in time to see what a particular area looked like in the past.
Of course, Scottsdale, my town, is not worth attempting this on, but older cities that have seen a ton of change will give some great inside into how neighborhoods have changed over the past century.
Street level view of 3D-reconstructed Chelsea, Manhattan
Just take a look at the image above, it really does give the feel of New York back in the ’40s and earlier. People remember how a neighborhood looked, but recreating it in this method gives others key insights into how development has changed how certain areas of cities look and act.
This tool is probably more aimed at history professors and community activists, but as we grow cities into smarter, cleaner places to live, understanding the past is how we can hope to create a better future. I’d love to see these tools be incorporated into smart city planning efforts. The great part of all this is it is crowdsourced, open-sourced, and worth doing. I’m starting to take a deeper dive into the GitHub repository and look how the output of this project can help plan better cities.
October 28, 2020 3D ai crowdsource google google ai open source planning Thoughts
Machine Learning Smart City Development from Sidewalk Labs
The last time most people heard from Sidewalk Labs was when Toronto didn’t go forward with their Smart City project. There are a ton of reasons why that didn’t happy, but moonshots are what they are and even if you don’t reach the moon, outcomes can be really good for society. Of course, I know not what Sidewalk Labs has been working on but I have to assume Delve exists because of the work they are doing to build smart cities.
Delve at its most simple description is where computers figure out the best design options for commercial or residential project development. there is much more going on here and that’s where the Machine Learning (ML) part comes in and what really catches my eye. I’ve done a tone of work with planning in my years of working with AECs and coming up with multiple design options is time-consuming and very difficult. But with Delve, this can happen quickly and repeatable in minutes.
A quick look at Delve
You get optimal design options based on ranked priority outcomes such as cost or view. Delve takes inputs such as zoning constraints (how high a building can be or what the setbacks are), gross floor area (commercial or residential), and then combines these with the priority outcomes. Then you get scored options that you can look further into and continue to make changes to the inputs.
The immediacy of this is what really sets this apart. When I was at Cityzenith years ago, we attempted to try and get this worked out but the ML tools were not developed enough yet. Clearly though, with Alphabet backing, Sidewalk Labs has created an amazing tool that will probably change how cities are being developed.
I am really excited to see how this works out. I don’t see an API yet so integration outside of Sidewalk labs does not seem to be a priority at this point but the outcome for scaleable planning like this needs to have an API. I’ll be paying attention but seeing ML being used for this type of development is logical, understandable, and workable. We should see great success. You can read more at the Sidewalk Labs blog.
October 27, 2020 design digital twin machine learning planning sidewalk labs smart cities Thoughts zoning
Developing a Method to Discover Assets Inside Digital Twins
On Monday I had a bit of a tweetstorm to get some thoughts on paper.
Thinking about addressing but inside buildings.
— James Fee (@jamesmfee) October 26, 2020
In there I laid out what I thought addressing inside a building should look like. A couple of responses came to the “why” or “this isn’t an issue” but the important thing here is with smart buildings, they need to be able to route people not only to offices for “business” but workers to IoT devices to act upon issues that might occur (like a water valve leaking in a utility closet). Sure one, could just pull out an as-built drawing and navigate, or in the case of visiting a company, the guard at the front door, but if things such as Apple Glass and Google Glass start becoming a real thing, we’ll need a true addressing system to get people where they need to be.
Apple and Google are working this out themselves inside their ecosystems but there needs to be an open standard that people can use inside their applications to share data. I mentioned Placekey as a good starting point with their what@where.
The what is an address - poi encoding and the where is based on Uber’s H3 system. As great as all this is, it doesn’t help us figure out where the leaky valve is in the utility closet. This all is much better than other systems and is a great way to get close. I’ve not seen any way to create extensions to Placekey to do this but we’ll punt the linking problem for now.
The other problem with addressing inside a building is the digital twin might not be in any projection that our maps understand. So we’ll need to create a custom grid to figure out where the IoT and other interesting features are located. But there seems to be a standard being created that solves just this problem, UBID.
UBID builds on the open-source grid reference system and is essentially the north axis-aligned “bounding box” of the building’s footprint represented as a centroid along with four cardinal extents.
I really like this, it might even compete with Placekey, but that’s not my battle, I’m more concerned with buildings in this use case. There is so much to UBID to digest and I encourage you to read the Github to learn more.
But if we can link these grids of buildings, with a Placekey, we have a superb method of navigating to a building POI and then drilling down into navigating to that location using all the great work that companies like Pixel8 are doing. But all that navigation stuff is not my battle, just a location of an IoT sensor in a digital twin that may or may not be in a project we can use.
Working toward that link, a unique grid of a digital twin to a Placekey would solve all problems with figuring out where an asset inside a building is and what is going on at that location. The ontologies to link this could open up whole new methods of interrogation of IoT devices and so much more. e911 and similar systems could greatly benefit from this as well.
October 27, 2020 Apple Glass digital twin Google Glass H3 iot pixel8 Placekey POI Thoughts uber UBID