Spatially Adjusted

Tag: digital twin

  • Book Review: Hands-On Azure Digital Twins: A practical guide to building distributed IoT solutions

    Finished reading: Hands-On Azure Digital Twins: A practical guide to building distributed IoT solutions by Alexander Meijers

    I’ve been working with Digital Twins for almost 10 years and as simple as the concept is, ontologies get in the way of implementations. The big deal is how can you implement, how does one actually create a digital twin and deploy it to your users.

    Let’s be honest though, the Azure Digital Twin service/product is complex and requires a ton of work. It isn’t an upload CAD drawing and connect some data sources. In this case Meijers does a great job of walking through how to get started. But it isn’t for beginners, you’ll need to have previous experience with Azure Cloud services, Microsoft Visual Studio and the ability to debug code. But if you’ve got even a general understand of this, the walk throughs are detailed enough to learn the idiosyncrasies of the Azure Digital Twin process.

    The book does take you through the process of understanding what an Azure Digital Twin model is, how to upload them, developing relationships between models and how to query them. After you have an understanding on this, Meijers dives into connecting the model to services, updating the Azure Digital Twin models and then connecting to Microsoft Azure Maps to view the model on maps. Finally he showcases how these Digital Twins can become smart buildings which is the hopeful outcome of doing all the work.

    The book has a lot of code examples and ability to download it all from a Github repository. Knowledge of JSON and JavaScript, Python and .NET or Java is probably required. BUT, even if you don’t know how to code, this book is a good introduction to Azure Digital Twins. While there are pages of code examples, Meijers does a good job of explaining the how and why you would use Azure Digital Twins. If you’re interesting in how you can use a hosted Digital Twin service that is managed by a cloud service, this is a great resource.

    I felt like I knew Azure Digital Twins before reading this book, but it taught me a lot about how and why Microsoft did what they did with the service. Many aspects that caused me to scratch my head became clearer to me and I felt like this book gave me additional background that I didn’t have before. This book requires an understanding of programming but after finishing it I felt like Meijers’ ability to describe the process outside of code makes the book well worth it to anyone who wants to understand the concept and architecture of Azure Digital Twins.

    Thoroughly enjoyed the book.

  • Unreal and Unity are the new Browsers

    Someone asked me why I hadn’t commented on Cesium and Unreal getting together. Honestly , no reason. This is big news honestly. HERE, where I work, is teaming up with Unity to bring the Unity SDK and the HERE SDK to automotive applications. I talk about how we used Mapbox Unity SDK at Cityzenith (though I have no clue if they still are). Google and Esri have them too. In fact both Unreal and Unity marketplaces are littered with data sources you can plug in.

    HERE Maps with Unity

    This is getting at the core of what these two platforms could be. Back in the day we had two browsers, Firefox and Internet Explorer 6. Inside each we had many choices of mapping platforms to use. From Google and Bing to Mapquest and Esri. In the end that competition to make the best API/SDK for a mapping environment drove a ton of innovation. What Google Maps looks like and does in 2021 vs 2005 is amazing.

    This brings up the key as to what I see happening here. We’ll see the mapping companies (or companies that have mapping APIs) deliver key updates to these SDK (which today are pretty limited in scope) because they have to stay relevant. Not that web mapping is going away at any point, but true 3D world and true Digital Twins require power that browsers cannot provide even in 2021. So this rush to become the Google Maps of 3D engines is real and will be fun to watch.

    Interesting in that Google is an also-ran in the 3D engine space, so there is so much opportunity for the players who have invested and continue to invest in these markets without Google throwing unlimited R&D dollars against it. Of course it only takes on press release to change all that so don’t bet against Google.

  • Digital Twins and Unreal Engine

    I’ve had a ton of experience with Unity and Digital Twins but I have been paying attention to Unreal Engine. I think the open nature of Unity is probably more suited for the current Digital Twin market, but competition is so important for innovation. This project where Unreal Engine was used to create a digital clone of Adelaide is striking but the article just leaves me wanting for so much more.

    A huge city environment results in a hefty 3D model. Having strategies in place to ease the load on your workstation is essential. “Twinmotion does not currently support dynamic loading of the level of detail, so in the case of Adelaide, we used high-resolution 3D model tiles over the CBD and merged them together,” says Marre. “We then merged a ring of low-resolution tiles around the CBD and used the lower level of detail tiles the further away we are from the CBD.”

    Well, that’s how we did it at Cityzenith. Tiles are the only way to have the detail one needs in these 3D worlds and one that geospatial practitioners are very used to dealing with their slippy maps. The eye-candy that one sees in that Adelaide project is amazing. Of course, scaling one city out is hard enough but doing so across a country or the globe is another. Still, this is an amazing start.

    Seeing Epic take Twinmotion and scale it out this way is very exciting because as you can see from that video above, it really does look photorealistic.

    But this gets at the core of where Digital Twins have failed. It is so very easy to do the above, crate an amazing looking model of a city, and drape imagery across it. It is a very different beast to actually create a Digital Twin where these buildings are not only linked up to external IoT devices and services but they should import BIM models and generalize as needed. They do so some rudimentary analysis of shadows which is somewhat interesting, but this kind of stuff is so easy to do and there are so many tools to do it that all this effort to create a photorealistic city seems wasted.

    I think users would trade photorealistic cities for detailed IoT services integration but I will watch Aerometrex continue to develop this out. Digital Twins are still stuck in sharing videos on Vimeo and YouTube, trying to create some amazing realistic city when all people want is visualization and analysis of IoT data. That said, Aerometrex has done an amazing job building this view.

  • Moving Towards a Digital Twin Ecosystem

    Smart Cities really start to become valuable when they integrate with Digital Twins. Smart Cities do really well with transportation networks and adjusting when things happen. Take, for example, construction on an important Interstate highway that connects the city core with the suburbs causes backups and a smart city can adjust traffic lights, rail, and other modes of transportation to help adjudicate the problems. This works really well because the transportation system talk to each other and decisions can be made to refocus commutes toward other modes of transportation or other routes. But unfortunately, Digital Twins don’t do a great job talking to Smart Cities.

    Photo by Victor Garcia on Unsplash

    A few months ago I talked about Digital Twins and messaging. The idea that:

    Digital twins require connectivity to work. A digital twin without messaging is just a hollow shell, it might as well be a PDF or a JPG. But connecting all the infrastructure of the real world up to a digital twin replicates the real world in a virtual environment. Networks collect data and store it in databases all over the place, sometimes these are SQL-based such as Postgres or Oracle, and other times they are simple as SQLite or flat-file text files. But data should be treated as messages back and forth between clients.

    This was in the context of a Digital Twin talking to services that might not be hardware-based, but the idea stands up for how and why a Digital Twin should be messaging the Smart City at large. Whatever benefits a Digital Twin gains from an ecosystem that collects and analyzes data for decision-making those benefits become multiplied when those systems connect to other Digital Twins. But think outside a group of Digital Twins and the benefit of the Smart City when all these buildings are talking to each other and the city to make better decisions about energy use, transportation, and other shared infrastructure across the city or even the region (where multiple Smart Cities talk to each other).

    When all these buildings talk to each other, they can help a city plan, grow and evolve into a clean city.

    What we don’t have is a common data environment (CDE) that cities can use. We have seen data sharing on a small scale in developments but not on a city-wide or regional scale. To do this we need to agree on model standards that allow not only Digital Twins to talk to each other (Something open like Bentley’s iTwin.js) and share ontologies. Then we need that Smart City CDE where data is shared, stored, and analyzed at a large scale.

    One great outcome of this CDE is all this data can be combined with City ordinances to give tools like Delve from Sidewalk Labs even more data to create their generative design options. Buildings are not a bubble in a city and their impacts on the city extend out beyond the boundaries of the parcel they are built on. That’s what so exciting about this opportunity, manage assets in a Digital Twin on a micro-scale, but share generalized data about those decisions to the city at large which then can share them with other Digital Twins.

    Graphic showing chart of change over time

    And lastly, individual Smart Cities aren’t bubbles either. They have huge impacts on the region or even the country that they are in. If we can figure out how to create a national CDE, one that covers a country as diverse as the United States, we can have something that can even benefit the world at large. Clean cities are the future and thinking about them on a small scale will only result in the gentrification of affluent areas and leave less well areas behind. I don’t want my children to grow up in a world like that and we have the processes in place to ensure that they have a better place than use to grow up in.

  • Apple’s Digital Twin is All About Augmented Reality

    Now before we get too far, Apple has not created anything close to a Digital Twin as we know them. But what they have done is created an easy way to import your building models into Apple Maps. Apple calls this their Indoor Maps program.

    Easily create detailed maps of your indoor spaces and let visitors see where they are right in your app. Organizations with large public and private spaces like airports, shopping centers, arenas, hospitals, universities, and private office buildings can register for the Indoor Maps Program. Indoor maps are built using industry standard tools and require only your existing Wi-Fi network to enable GPS-level location accuracy so visitors can navigate your spaces with ease.

    Victoria Airport in the Apple IMDF Sandbox

    OK, so clearly this is all about navigation. How do I know where I am in a building and how do I get to a place I need to be. Of course, this is somewhat interesting on your iPhone or iPad in Apple Maps, but clearly, there is more to this than just how do I find the restroom on floor 10 of the bank tower.

    To load your buildings in Apple you need to use Mapkit or Mapkit.js and convert your buildings into Indoor Mapping Data Format (IMDF). IMDF is actually a great choice because it is GeoJSON and working toward being an OGC standard (for whatever that is worth these days). I did find it interesting that Apple highlights the following as the use case for IMDF:

    • Indoor wayfinding
    • Indoor routing
    • Temporal constraints
    • Connectivity amongst mapped objects
    • Location-based services
    • Query and find by location functionality

    If you’re familiar with GeoJSON, IMDF will look logical to you:

    {
  "id": "11111111-1111-1111-1111-111111111111",
  "type": "Feature",
  "feature_type": "building",
  "geometry": null,
  "properties": {
    "category": "parking",
    "restriction": "employeesonly",
    "name": {
      "en": "Parking Garage 1"
    },
    "alt_name": null,
    "display_point": {
      "type": "Point",
      "coordinates": [1.0, 2.0]
    },
    "address_id": "2222222-2222-2222-2222-222222222222"
  }
}

    I encourage you to review the IMDF docs to learn more but we’re talking JSON here so it’s exactly how you’d expect it to work.

    Because IMDF buildings are generalized representations of the real-world data, this isn’t actually a Digital Twin. It also means that you need to do some things to your files before converting them to IMDF. Autodesk, Esri, and Safe Software all support IMDF so you should be able to use their tools to handle the conversions. I’ve done the conversion with Safe FME and it works very well and probably the best way to handle this. In fact, Safe has an IMDF validator which does come in handy for sure.

    Safe FME support of IMDF

    What does make moving your buildings to Apple’s Indoor platform is the new iPhone 12 and iPad Pro LiDAR support. This brings out some really great AR capabilities that become enabled with Apple’s devices. As I said last week, the LiDAR support in the current devices is more about getting experience with LiDAR use cases than actual LiDAR use. This is all about eventual Apple Glass (and Google Glass too) support and the AR navigation that can be done when you have hyper-accurate indoor models in your mapping software.

    I’ve been dusting off my MapKit skills because I think not only is this capability useful for many companies but it really isn’t that hard to enable. I am spending some time thinking about how to use the extension capability of IMDF to see how IoT and other services can be brought in. Given the generalized nature of IMDF, it could be a great way to allow visualizing IoT and other services without the features of a building getting in the way. Stay tuned!

  • The iPhone 12 Pro LiDAR Scanner is the Gateway to AR, But Not in the Way You Think

    I’m sure everyone knows about it by now, the iPhone 12 Pro has a LiDAR scanner. Apple touts it to help you take better pictures in low light and do some rudimentary AR on the iPhone. But, what this scanner does today isn’t where the power will be tomorrow.

    Apple cares a ton about photo quality, so a LiDAR scanner helps immensely with taking these pictures. If there is one reason today to have that scanner, it is for pictures. But the real power of the scanner is for AR. And AR isn’t ready today, no matter how many demos you see in Apple’s event. Holding up an iPhone and seeing how big a couch in your room is interesting, just as interesting as using your phone to find the nearest Starbucks.

    Apple has spent a lot of time working on interior spaces in Apple Maps. They’ve also spent a ton of time working on sensors in the phone for positioning inside buildings. This is all building to an AR navigation space inside public buildings and private buildings in which owners share their 3D plans. But what if hundreds of millions of mobile devices could create these 3D worlds automatically as they go about their business helping users find that Starbucks?

    The future is so bright though with this scanner. It helps Apple and developers get familiar with what LiDAR can do for AR applications. This is critically important on the hardware side because Apple Glass, no matter how little is known about it, is the future for AR. Same with Google Glass too, the eventual consumer product (ignoring the junk that the first Google Glass was) of these wearable AR devices will change the world, not so much in that you’ll see an arrow as you navigate to the Starbucks, but give you the insight into smart buildings and all the IoT devices that are around.

    The inevitable outcome is in the maintenance of smart buildings

    Digital Twins are valuable when they link data feeds to a 3D world that can be interrogated. But the real value comes when those 3D worlds can be leveraged using Augmented Reality to give owners, maintenance workers, planners, engineers, and tenants the information they need to service their buildings and improve the quality of building maintenance. The best built LEED building is only as good as the ongoing maintenance put on it.

    The iPhone 12 Pro and the iPad Pro that Apple has released this year both have LiDAR to improve their use with photo taking and rudimentary AR, but the experience gained seeing the real-world use of consumer LiDAR in millions of devices will bring great strides to making these Apple/Google Glass devices truly usable in real-world use. I’m still waiting to get my iPhone 12, but my wife’s arrived today. I’m looking forward to seeing what the LiDAR can do.

  • Developing a Method to Discover Assets Inside Digital Twins

    On Monday I had a bit of a tweetstorm to get some thoughts on paper.

    In there I laid out what I thought addressing inside a building should look like. A couple of responses came to the “why” or “this isn’t an issue” but the important thing here is with smart buildings, they need to be able to route people not only to offices for “business” but workers to IoT devices to act upon issues that might occur (like a water valve leaking in a utility closet). Sure one, could just pull out an as-built drawing and navigate, or in the case of visiting a company, the guard at the front door, but if things such as Apple Glass and Google Glass start becoming a real thing, we’ll need a true addressing system to get people where they need to be.

    Apple and Google are working this out themselves inside their ecosystems but there needs to be an open standard that people can use inside their applications to share data. I mentioned Placekey as a good starting point with their what@where.

    The what is an address – poi encoding and the where is based on Uber’s H3 system. As great as all this is, it doesn’t help us figure out where the leaky valve is in the utility closet. This all is much better than other systems and is a great way to get close. I’ve not seen any way to create extensions to Placekey to do this but we’ll punt the linking problem for now.

    The other problem with addressing inside a building is the digital twin might not be in any projection that our maps understand. So we’ll need to create a custom grid to figure out where the IoT and other interesting features are located. But there seems to be a standard being created that solves just this problem, UBID.

    UBID builds on the open-source grid reference system and is essentially the north axis-aligned “bounding box” of the building’s footprint represented as a centroid along with four cardinal extents.

    I really like this, it might even compete with Placekey, but that’s not my battle, I’m more concerned with buildings in this use case. There is so much to UBID to digest and I encourage you to read the Github to learn more.

    But if we can link these grids of buildings, with a Placekey, we have a superb method of navigating to a building POI and then drilling down into navigating to that location using all the great work that companies like Pixel8 are doing. But all that navigation stuff is not my battle, just a location of an IoT sensor in a digital twin that may or may not be in a project we can use.

    Working toward that link, a unique grid of a digital twin to a Placekey would solve all problems with figuring out where an asset inside a building is and what is going on at that location. The ontologies to link this could open up whole new methods of interrogation of IoT devices and so much more. e911 and similar systems could greatly benefit from this as well.

  • Machine Learning Smart City Development from Sidewalk Labs

    The last time most people heard from Sidewalk Labs was when Toronto didn’t go forward with their Smart City project. There are a ton of reasons why that didn’t happy, but moonshots are what they are and even if you don’t reach the moon, outcomes can be really good for society. Of course, I know not what Sidewalk Labs has been working on but I have to assume Delve exists because of the work they are doing to build smart cities.

    Delve at its most simple description is where computers figure out the best design options for commercial or residential project development. there is much more going on here and that’s where the Machine Learning (ML) part comes in and what really catches my eye. I’ve done a tone of work with planning in my years of working with AECs and coming up with multiple design options is time-consuming and very difficult. But with Delve, this can happen quickly and repeatable in minutes.

    A quick look at Delve

    You get optimal design options based on ranked priority outcomes such as cost or view. Delve takes inputs such as zoning constraints (how high a building can be or what the setbacks are), gross floor area (commercial or residential), and then combines these with the priority outcomes. Then you get scored options that you can look further into and continue to make changes to the inputs.

    The immediacy of this is what really sets this apart. When I was at Cityzenith years ago, we attempted to try and get this worked out but the ML tools were not developed enough yet. Clearly though, with Alphabet backing, Sidewalk Labs has created an amazing tool that will probably change how cities are being developed.

    I am really excited to see how this works out. I don’t see an API yet so integration outside of Sidewalk labs does not seem to be a priority at this point but the outcome for scaleable planning like this needs to have an API. I’ll be paying attention but seeing ML being used for this type of development is logical, understandable, and workable. We should see great success. You can read more at the Sidewalk Labs blog.

  • Capturing As-Built Changes to Make Better Digital Twins

    This post originally appeared on LinkedIn.
    Augmented Reality view of Apple Park

    Digital Twins are easy. All you have to do is create a 3D object. Some triangles and you’re done. A BIM model is practically a Digital Twin. The problem is usually those twins are created from data that isn’t “as-built“. What you end up with is a digital object that ISN’T a twin. How can you connect your IoT and other assets to a 3D object that isn’t representative of the real world?

    I talked a little bit last time on how to programmatically create digital twins from satellite and other imagery. Of course, a good constellation can make these twins very up to date and accurate but it can miss the details needed for a good twin and it sure as heck can’t get inside a building to update any changes there. What we’re looking for here is a feedback loop, from design to construction to digital twin.

    There are a lot of companies that can help with this process so I won’t go into detail there, but what is needed is the acknowledgment that investment is needed to make sure those digital twins are updated, not only is the building being delivered but an accurate BIM model that can be used as a digital twin. Construction firms usually don’t get the money to update these BIM models so they are used as a reference at the beginning, but change orders rarely get pushed back to the original BIM models provided by the architects. That said there are many methods that can be used to close this loop.

    Construction methods cause changes from the architectural plans

    Companies such as Pixel8 that I talked about last week can use high-resolution imagery and drones to create a point cloud that can be used to verify not only changes are being made as specifications but also can notify where deviations have been made from the BIM model. This is big because humans can only see so much on a building, and with a large model, it is virtually impossible for people to detect change. But using machine learning and point clouds, change detection is actually very simple and can highlight where accepted modifications have been made to the architectural drawings or where things have gone wrong.

    The key point here is using ML to discover and update digital twins at scale is critically important, but just as important is the ability to use ML to discover and update digital twins as they are built, rather than something that came from paper space.

    Credits:

    Photo by Patrick Schneider on Unsplash
    Photo by Elmarie van Rooyen on Unsplash
    Photo by Scott Blake on Unsplash

  • Scaling Digital Twins

    This article originally appeared on LinkedIn.

    Let’s face it, digital twins make sense and there is no arguing their purpose. At least with the urban landscape though, it is very difficult to scale digital twins beyond a section of a city. At Cityzenith we attempted to overcome this need to have 3D buildings all over the world and used a 3rd party service that basically extruded OSM building footprints where they existed. You see this in the Microsoft Flight Simulator worlds that people have been sharing, it looks pretty good from a distance but up close it becomes clear that building footprints are a horrible way to represent a digital twin of the built environment because they are so inaccurate. Every building is not a rectangle and it becomes impossible to perform any analysis on them because they can be off upwards of 300% on their real-world structure.

    Microsoft Flights Simulator created world-wide digital twins at a very rough scale.

    How do you scale this problem, creating 3D buildings for EVERYWHERE? Even Google was unable to do this, they tried to get people to create accurate 3D buildings with Sketchup but that failed, and they tossed the product over to Trimble where it has gotten back to its roots in the AEC space. If Google can’t do it who can?

    Vricon, who was a JV between Maxar and Saab but recently absorbed by Maxar completely, gives a picture into how this can be done. Being able to identify buildings, extract their shape, drape imagery over them, and then continue to monitor change over the years as additions, renovations, and even rooftop changes are identified. There is no other way I can see that we can have worldwide digital twins other than using satellite imagery.

    Vricon is uniquely positioned to create on demand Digital Twins world-wide.

    Companies such as Pixel8 also play a part in this. I’ve already talked about how this can be accomplished on my blog; I encourage you to take a quick read on it. The combination of satellite digital twins to cover the world and then using products such as Pixel8 can create that highly detailed ground truth that is needed in urban areas. In the end, you get an up to date, highly accurate 3D model that actually allows detailed analysis of impacts from new buildings or other disruptive changes in cities.

    Hyper-accurate point clouds from imagery, hand-held or via drone.

    But to scale out a complete digital twin of the world at scale, the only way to accomplish this is through satellite imagery. Maxar and others are already using ML to find buildings and discover if they have changed over time. Coupled with the technology that Vricon brings inside Maxar, I can see them really jump-starting a service of worldwide digital twins. Imagine being able to bring accurate building models into your analysis or products that not only are hyper-accurate compared to extruded footprints but are updated regularly based on the satellite imagery collected.

    That sounds like the perfect world, Digital Twins as a Service.