Research engineers at the University of Waterloo have published a paper documenting their use of AR, VR and AI to inspect civil infrastructure such as bridges and enable close collaboration between teams of inspectors onsite and off site.
Waterloo professor Chul Min Yeum and his colleagues have developed a system called the Smart Infrastructure Metaverse (SIM) that allows the two teams of inspectors to view the same hologram depicting structural deficiencies such as spalling.
The teams of inspectors can interact and share information, creating faster, better and more complete reports and providing greater context for analyzing problems, Yeum says.
Data is gathered by a robot and processed by a machine learning detection algorithm that automatically detects and tracks visual defects in the structure.
Onsite, off-site teams collaborate
“The Smart Infrastructure Metaverse is all about making it easier for onsite and off-site inspectors to work together on structural inspections,” Yeum said.
“We’re using AR/VR headsets to give them a shared view so they can see exactly where they are and what they’re looking at during the inspection, no matter where they’re located.”
Yeum said until now there has been a technical gap that made it challenging to adequately determine the location of inspectors onsite. Visual inspections may have constraints when dealing with large, complex structures or environments that are difficult to access.
As a remote research partner, Yeum said, “I want to see more on the specific area. But in that case, there is a person at their own site. How can I tell them to look at the specific area in more detail?”
The Waterloo researchers are developing an image-based localization algorithm called Multi-Shot Localization to align panoramic images in a 3D map to provide an all-around view of the infrastructure site.
Tested at Kitchener bridge
Yeum and his colleagues tested their proposals during an inspection of a railway bridge in Kitchener. They created a three-dimensional model of the bridge using 3D scanners and a panorama camera. The hologram model allowed accurate tracking of the location and position of both the onsite and off-site inspectors within the 3D map.
Onsite inspectors using AR (or technically MR, mixed AR and VR) headsets worked with remote experts using VR.
“The MR inspectors execute inspection tasks such as adding annotations or taking measurements via MR interfaces, while the VR inspectors also perform the same tasks in a virtual environment,” Yeum’s paper, published in September, explained.
The off-site inspector saw the exact locations of the onsite users along with the areas of the structure they were inspecting.
“There is the real-time interacting,” said Yeum. “They can see where they are and what they are looking at. And also we can see from their gestures where their finger is heading to, so that people can see the area, and at the same time there’s communication.”
In addition, the research team used AI to analyze the images sent from the onsite inspectors’ AR headsets to more deeply analyze structural damage.
Yeum’s research has been financed by Rogers since 2020, with the technology being put into place to inspect cell towers. As for commercialization in the infrastructure inspection sector, he figures it might be several years, perhaps five, away from hitting the market.
The biggest bottleneck, Yeum said, is the quality of the equipment, its accuracy, and its expense. Current equipment is cumbersome to use.
He sees genuine interest in the construction sector in the system and there has been communication with Ontario’s Ministry of Transportation.
A video explaining the system notes humans remain an important part of the process.
“We hope to provide inspectors with a solution that improves inspection outcomes, such as increased accuracy and efficiency, as well as increased collaborations between humans and technology,” the video states.
Recent Comments