Apple’s Digital Twin is All About Augmented Reality

Now before we get too far, Apple has not created anything close to a Digital Twin as we know them. But what they have done is created an easy way to import your building models into Apple Maps. Apple calls this their Indoor Maps program.

Easily create detailed maps of your indoor spaces and let visitors see where they are right in your app. Organizations with large public and private spaces like airports, shopping centers, arenas, hospitals, universities, and private office buildings can register for the Indoor Maps Program. Indoor maps are built using industry standard tools and require only your existing Wi-Fi network to enable GPS-level location accuracy so visitors can navigate your spaces with ease.

Victoria Airport in the Apple IMDF Sandbox

OK, so clearly this is all about navigation. How do I know where I am in a building and how do I get to a place I need to be. Of course, this is somewhat interesting on your iPhone or iPad in Apple Maps, but clearly, there is more to this than just how do I find the restroom on floor 10 of the bank tower.

To load your buildings in Apple you need to use Mapkit or Mapkit.js and convert your buildings into Indoor Mapping Data Format (IMDF). IMDF is actually a great choice because it is GeoJSON and working toward being an OGC standard (for whatever that is worth these days). I did find it interesting that Apple highlights the following as the use case for IMDF:

Indoor wayfinding
Indoor routing
Temporal constraints
Connectivity amongst mapped objects
Location-based services
Query and find by location functionality

If you’re familiar with GeoJSON, IMDF will look logical to you:

{
  "id": "11111111-1111-1111-1111-111111111111",
  "type": "Feature",
  "feature_type": "building",
  "geometry": null,
  "properties": {
    "category": "parking",
    "restriction": "employeesonly",
    "name": {
      "en": "Parking Garage 1"
    },
    "alt_name": null,
    "display_point": {
      "type": "Point",
      "coordinates": [1.0, 2.0]
    },
    "address_id": "2222222-2222-2222-2222-222222222222"
  }
}

I encourage you to review the IMDF docs to learn more but we’re talking JSON here so it’s exactly how you’d expect it to work.

Because IMDF buildings are generalized representations of the real-world data, this isn’t actually a Digital Twin. It also means that you need to do some things to your files before converting them to IMDF. Autodesk, Esri, and Safe Software all support IMDF so you should be able to use their tools to handle the conversions. I’ve done the conversion with Safe FME and it works very well and probably the best way to handle this. In fact, Safe has an IMDF validator which does come in handy for sure.

What does make moving your buildings to Apple’s Indoor platform is the new iPhone 12 and iPad Pro LiDAR support. This brings out some really great AR capabilities that become enabled with Apple’s devices. As I said last week, the LiDAR support in the current devices is more about getting experience with LiDAR use cases than actual LiDAR use. This is all about eventual Apple Glass (and Google Glass too) support and the AR navigation that can be done when you have hyper-accurate indoor models in your mapping software.

I’ve been dusting off my MapKit skills because I think not only is this capability useful for many companies but it really isn’t that hard to enable. I am spending some time thinking about how to use the extension capability of IMDF to see how IoT and other services can be brought in. Given the generalized nature of IMDF, it could be a great way to allow visualizing IoT and other services without the features of a building getting in the way. Stay tuned!

COVID-19 is Showing How Smart Cities Protect Citizens

I feel like there is a before COVID and an after COVID with citizens’ feelings for Smart City technology. Now there is an election tomorrow in the United States that will probably dictate how this all moves forward and after 2016, I’ve learned to not predict anything when it comes to the current president. But, outside that huge elephant in the background, Smart City concepts have been thrust into the spotlight.

Most cities have sent their non-essential workers home, so IoT and other feeds to their work dashboards have become critical to their success. The data collection and analysis of the pulse of a city is now so important that traditional field collection tools have become outdated.

Even how cities engage with their citizens has changed. Before COVID, here in Scottsdale, you needed to head to a library to get a library card in person. But since COVID restrictions, the city has allowed library card applications in person which is a huge change. The core structure of city digital infrastructure has to change to manage this new need. Not only engaging citizens deeper with technology but need to ensure those who don’t have access to the internet or even a computer are represented. I’ve seen much better smartphone access on websites over the summer and this will continue.

Even moving from a public space to a digital space for city council meetings has implications. The physicality of citizens before their elected leaders is a check on their power, but being a small zoom box in a monitor of zoom boxes puts citizens in a corner. Much will have to be developed to have a way for those who don’t wish to be in person be represented as well as those who choose to attend meetings in person.

COVID has also broken down barriers to sharing data. The imagined dashboard where Police, Fire, Parks & Rec, City Council, and other stakeholders have come to fruition. The single pane of glass where decision-makers can get together to run the city remotely is only going to improve now that the value has been shown.

Lastly, ignoring the possible election tomorrow, contact tracing, and other methods of monitoring citizens as they go around the city has changed mostly how people feel. Before COVID, the idea that a city could track them even anonymously scared the daylights out of people. But today we are starting to see the value in anonymous tracking so that not only we see who has been in contact with each other but how they interact in a city with social distancing restrictions.

Future planning of cities is changing and accelerated because of COVID. The outcome of this pandemic will result in cities that are more resilient, better managed, planned for social distancing, and are working toward carbon neutral environments. In the despair of this unprecedented pandemic, we see humanity coming together to create a better future for our cities and our planet.

The iPhone 12 Pro LiDAR Scanner is the Gateway to AR, But Not in the Way You Think

I’m sure everyone knows about it by now, the iPhone 12 Pro has a LiDAR scanner. Apple touts it to help you take better pictures in low light and do some rudimentary AR on the iPhone. But, what this scanner does today isn’t where the power will be tomorrow.

Apple cares a ton about photo quality, so a LiDAR scanner helps immensely with taking these pictures. If there is one reason today to have that scanner, it is for pictures. But the real power of the scanner is for AR. And AR isn’t ready today, no matter how many demos you see in Apple’s event. Holding up an iPhone and seeing how big a couch in your room is interesting, just as interesting as using your phone to find the nearest Starbucks.

Apple has spent a lot of time working on interior spaces in Apple Maps. They’ve also spent a ton of time working on sensors in the phone for positioning inside buildings. This is all building to an AR navigation space inside public buildings and private buildings in which owners share their 3D plans. But what if hundreds of millions of mobile devices could create these 3D worlds automatically as they go about their business helping users find that Starbucks?

The future is so bright though with this scanner. It helps Apple and developers get familiar with what LiDAR can do for AR applications. This is critically important on the hardware side because Apple Glass, no matter how little is known about it, is the future for AR. Same with Google Glass too, the eventual consumer product (ignoring the junk that the first Google Glass was) of these wearable AR devices will change the world, not so much in that you’ll see an arrow as you navigate to the Starbucks, but give you the insight into smart buildings and all the IoT devices that are around.

The inevitable outcome is in the maintenance of smart buildings

Digital Twins are valuable when they link data feeds to a 3D world that can be interrogated. But the real value comes when those 3D worlds can be leveraged using Augmented Reality to give owners, maintenance workers, planners, engineers, and tenants the information they need to service their buildings and improve the quality of building maintenance. The best built LEED building is only as good as the ongoing maintenance put on it.

The iPhone 12 Pro and the iPad Pro that Apple has released this year both have LiDAR to improve their use with photo taking and rudimentary AR, but the experience gained seeing the real-world use of consumer LiDAR in millions of devices will bring great strides to making these Apple/Google Glass devices truly usable in real-world use. I’m still waiting to get my iPhone 12, but my wife’s arrived today. I’m looking forward to seeing what the LiDAR can do.

Google AI Project Recreating Historical Streetscapes in 3D

When this caught my eye I got really interested. Google AI is launching a website titled rǝ which reconstructs cities from historical maps and photos. You might have seen the underlying tool last month but this productizes it a bit. What I find compelling about this effort is the output is a 3D city that you can navigate and review by going in back in time to see what a particular area looked like in the past.

Of course, Scottsdale, my town, is not worth attempting this on, but older cities that have seen a ton of change will give some great inside into how neighborhoods have changed over the past century.

Street level view of 3D-reconstructed Chelsea, Manhattan

Just take a look at the image above, it really does give the feel of New York back in the ’40s and earlier. People remember how a neighborhood looked, but recreating it in this method gives others key insights into how development has changed how certain areas of cities look and act.

This tool is probably more aimed at history professors and community activists, but as we grow cities into smarter, cleaner places to live, understanding the past is how we can hope to create a better future. I’d love to see these tools be incorporated into smart city planning efforts. The great part of all this is it is crowdsourced, open-sourced, and worth doing. I’m starting to take a deeper dive into the GitHub repository and look how the output of this project can help plan better cities.

Developing a Method to Discover Assets Inside Digital Twins

On Monday I had a bit of a tweetstorm to get some thoughts on paper.

Thinking about addressing but inside buildings.
— Captain Lammers (@cageyjames) October 26, 2020

In there I laid out what I thought addressing inside a building should look like. A couple of responses came to the “why” or “this isn’t an issue” but the important thing here is with smart buildings, they need to be able to route people not only to offices for “business” but workers to IoT devices to act upon issues that might occur (like a water valve leaking in a utility closet). Sure one, could just pull out an as-built drawing and navigate, or in the case of visiting a company, the guard at the front door, but if things such as Apple Glass and Google Glass start becoming a real thing, we’ll need a true addressing system to get people where they need to be.

Apple and Google are working this out themselves inside their ecosystems but there needs to be an open standard that people can use inside their applications to share data. I mentioned Placekey as a good starting point with their what@where.

The what is an address – poi encoding and the where is based on Uber’s H3 system. As great as all this is, it doesn’t help us figure out where the leaky valve is in the utility closet. This all is much better than other systems and is a great way to get close. I’ve not seen any way to create extensions to Placekey to do this but we’ll punt the linking problem for now.

The other problem with addressing inside a building is the digital twin might not be in any projection that our maps understand. So we’ll need to create a custom grid to figure out where the IoT and other interesting features are located. But there seems to be a standard being created that solves just this problem, UBID.

UBID builds on the open-source grid reference system and is essentially the north axis-aligned “bounding box” of the building’s footprint represented as a centroid along with four cardinal extents.

I really like this, it might even compete with Placekey, but that’s not my battle, I’m more concerned with buildings in this use case. There is so much to UBID to digest and I encourage you to read the Github to learn more.

But if we can link these grids of buildings, with a Placekey, we have a superb method of navigating to a building POI and then drilling down into navigating to that location using all the great work that companies like Pixel8 are doing. But all that navigation stuff is not my battle, just a location of an IoT sensor in a digital twin that may or may not be in a project we can use.

Working toward that link, a unique grid of a digital twin to a Placekey would solve all problems with figuring out where an asset inside a building is and what is going on at that location. The ontologies to link this could open up whole new methods of interrogation of IoT devices and so much more. e911 and similar systems could greatly benefit from this as well.

Machine Learning Smart City Development from Sidewalk Labs

The last time most people heard from Sidewalk Labs was when Toronto didn’t go forward with their Smart City project. There are a ton of reasons why that didn’t happy, but moonshots are what they are and even if you don’t reach the moon, outcomes can be really good for society. Of course, I know not what Sidewalk Labs has been working on but I have to assume Delve exists because of the work they are doing to build smart cities.

Delve at its most simple description is where computers figure out the best design options for commercial or residential project development. there is much more going on here and that’s where the Machine Learning (ML) part comes in and what really catches my eye. I’ve done a tone of work with planning in my years of working with AECs and coming up with multiple design options is time-consuming and very difficult. But with Delve, this can happen quickly and repeatable in minutes.

You get optimal design options based on ranked priority outcomes such as cost or view. Delve takes inputs such as zoning constraints (how high a building can be or what the setbacks are), gross floor area (commercial or residential), and then combines these with the priority outcomes. Then you get scored options that you can look further into and continue to make changes to the inputs.

The immediacy of this is what really sets this apart. When I was at Cityzenith years ago, we attempted to try and get this worked out but the ML tools were not developed enough yet. Clearly though, with Alphabet backing, Sidewalk Labs has created an amazing tool that will probably change how cities are being developed.

I am really excited to see how this works out. I don’t see an API yet so integration outside of Sidewalk labs does not seem to be a priority at this point but the outcome for scaleable planning like this needs to have an API. I’ll be paying attention but seeing ML being used for this type of development is logical, understandable, and workable. We should see great success. You can read more at the Sidewalk Labs blog.

Capturing As-Built Changes to Make Better Digital Twins

This post originally appeared on LinkedIn.

Digital Twins are easy. All you have to do is create a 3D object. Some triangles and you’re done. A BIM model is practically a Digital Twin. The problem is usually those twins are created from data that isn’t “as-built“. What you end up with is a digital object that ISN’T a twin. How can you connect your IoT and other assets to a 3D object that isn’t representative of the real world?

I talked a little bit last time on how to programmatically create digital twins from satellite and other imagery. Of course, a good constellation can make these twins very up to date and accurate but it can miss the details needed for a good twin and it sure as heck can’t get inside a building to update any changes there. What we’re looking for here is a feedback loop, from design to construction to digital twin.

There are a lot of companies that can help with this process so I won’t go into detail there, but what is needed is the acknowledgment that investment is needed to make sure those digital twins are updated, not only is the building being delivered but an accurate BIM model that can be used as a digital twin. Construction firms usually don’t get the money to update these BIM models so they are used as a reference at the beginning, but change orders rarely get pushed back to the original BIM models provided by the architects. That said there are many methods that can be used to close this loop.

Construction methods cause changes from the architectural plans

Companies such as Pixel8 that I talked about last week can use high-resolution imagery and drones to create a point cloud that can be used to verify not only changes are being made as specifications but also can notify where deviations have been made from the BIM model. This is big because humans can only see so much on a building, and with a large model, it is virtually impossible for people to detect change. But using machine learning and point clouds, change detection is actually very simple and can highlight where accepted modifications have been made to the architectural drawings or where things have gone wrong.

The key point here is using ML to discover and update digital twins at scale is critically important, but just as important is the ability to use ML to discover and update digital twins as they are built, rather than something that came from paper space.

Credits:

Photo by Patrick Schneider on Unsplash
Photo by Elmarie van Rooyen on Unsplash
Photo by Scott Blake on Unsplash

Scaling Digital Twins

This article originally appeared on LinkedIn.

Let’s face it, digital twins make sense and there is no arguing their purpose. At least with the urban landscape though, it is very difficult to scale digital twins beyond a section of a city. At Cityzenith we attempted to overcome this need to have 3D buildings all over the world and used a 3rd party service that basically extruded OSM building footprints where they existed. You see this in the Microsoft Flight Simulator worlds that people have been sharing, it looks pretty good from a distance but up close it becomes clear that building footprints are a horrible way to represent a digital twin of the built environment because they are so inaccurate. Every building is not a rectangle and it becomes impossible to perform any analysis on them because they can be off upwards of 300% on their real-world structure.

Microsoft Flights Simulator created world-wide digital twins at a very rough scale.

How do you scale this problem, creating 3D buildings for EVERYWHERE? Even Google was unable to do this, they tried to get people to create accurate 3D buildings with Sketchup but that failed, and they tossed the product over to Trimble where it has gotten back to its roots in the AEC space. If Google can’t do it who can?

Vricon, who was a JV between Maxar and Saab but recently absorbed by Maxar completely, gives a picture into how this can be done. Being able to identify buildings, extract their shape, drape imagery over them, and then continue to monitor change over the years as additions, renovations, and even rooftop changes are identified. There is no other way I can see that we can have worldwide digital twins other than using satellite imagery.

Vricon is uniquely positioned to create on demand Digital Twins world-wide.

Companies such as Pixel8 also play a part in this. I’ve already talked about how this can be accomplished on my blog; I encourage you to take a quick read on it. The combination of satellite digital twins to cover the world and then using products such as Pixel8 can create that highly detailed ground truth that is needed in urban areas. In the end, you get an up to date, highly accurate 3D model that actually allows detailed analysis of impacts from new buildings or other disruptive changes in cities.

Hyper-accurate point clouds from imagery, hand-held or via drone.

But to scale out a complete digital twin of the world at scale, the only way to accomplish this is through satellite imagery. Maxar and others are already using ML to find buildings and discover if they have changed over time. Coupled with the technology that Vricon brings inside Maxar, I can see them really jump-starting a service of worldwide digital twins. Imagine being able to bring accurate building models into your analysis or products that not only are hyper-accurate compared to extruded footprints but are updated regularly based on the satellite imagery collected.

That sounds like the perfect world, Digital Twins as a Service.

IoT is not About Hardware

When you think about IoT you think about little devices everywhere doing their own thing. From Nest thermostats and Ring doorbells to Honeywell environmental controls and Thales biometrics; you imagine hardware. Sure, there is the “I” part of IoT that conveys some sort of digital aspect, but we imagine the “things” part. But the simple truth of IoT is the hardware is a commodity and the true power in IoT is in the “I” part or the messaging.

IoT messages can inundate us but they are the true power of these sensors

IoT messages are usually HTTP, WebSockets, and MQTT or some derivative of them. MQTT is the one that I’m always most interested, but anything works which is what is so great about IoT as a service. MQTT is leveraged greatly by AWS IoT and Azure IoT and both services work so well at messaging that you can use either in replacement of something like RabbitMQ, which my daughter loves because of the rabbit icon. I could write a whole post on MQTT but we’ll leave that for another day.

IoT itself is built upon this messaging. That individual hardware devices have UIDs (unique identifiers) that by their very nature allow them to be unique. The packets of information that are sent back and forth between the device and the host are short messages that require no human interaction.

The best part of this is that you don’t need to hardware for IoT. Everything that you want to interact with should be an IoT message, no matter if it is an email, data query or text message. Looking at IoT as more than just hardware opens connectivity opportunities that had been much harder in the past.

Digital twins require connectivity to work. A digital twin without messaging is just a hollow shell, it might as well be a PDF or a JPG. But connecting all the infrastructure of the real world up to a digital twin replicates the real world in a virtual environment. Networks collect data and store it in databases all over the place, sometimes these are SQL-based such as Postgres or Oracle and other times they are simple as SQLite or flat file text files. But data should be treated as messages back and forth between clients.

When I look at the world, I see messaging opportunities, how we connect devices between each other. Seeing the world this way allows new ways to bring in data to Digital Twins, think of GIS services being IoT devices, and much easier ways to get more out of your digital investments.

Open Environments and Digital Twins

The GIS world has no idea how hard it is to work with data in the digital twin/BIM world. Most GIS formats are open, or at works readable to import into a closed system. But in the digital twin/BIM space, there is too many close data sets that makes it so hard to work with the data. The loops one must go through to import a Revit model are legendary and mostly are how you get your data into IFC without giving up all the intelligence. At Cityzenith, we were able to work with tons of open formats, but dealing with Revit and other closed formats was very difficult to the point it required a team in India to handle the conversions.

All the above is maddening because if there is one thing a digital twin should do, is be able to talk with as many other systems as possible. IoT messages, GIS datasets, APIs galore and good old fashioned CAD systems. That’s why open source data formats are best, those that are understood and can be extended in any way someone needs. One of the biggest formats that we worked with was glTF. It is widely supported these days but it really isn’t a great format for BIM models or other digital twin layers because it is more of a visual format than a data storage model. Think of it similar to a JPEG, great for final products, but you don’t want to work with it for your production data.

IFC, which I mentioned before, is basically a open BIM standard. IFC is actually a great format for BIM, but companies such as Autodesk don’t do a great job supporting it, it becomes more of interchange file, except where governments require it’s use. I also dislike the format because it is unwieldy, but it does a great job of interoperability and is well supported by many platforms.

IFC and GLTF are great, but they harken back to older format structures. They don’t take advantage of modern cloud based systems. I’ve been looking at DTDL (Digital Twins Definition Language) from Microsoft. What I do like about DLDT is that it is based on JSON-LD so many of those IoT services you are already working with take advantage of it. Microsoft’s Digital Twin platform was slow to take off but many companies, including Bentley Systems, are leveraging it to help their customers get a cloud based open platform which is what they all want. Plus you can use services such as Azure Functions (very underrated service IMO) to work with your data once it is in there.

The magic of digital twins is when you can connect messaging (IoT) services to your digital models. That’s the holy grail, have the real world connected to the digital world. Sadly, most BIM and digital twin systems aren’t open enough and require custom conversion work or custom coding to enable even simple integration with SAP, Salesforce or MAXIMO. That’s why these newer formats, based mostly on JSON, seem to fit the bill and we will see exponential growth in their use.