All you need to know about Holoportation with Mesh

We hear about advances in mixed reality. But the new wave capturing attention is holoportation—a 3D capture technology beaming a lifelike image of a person into a virtual scene. Microsoft recently, in the keynote let audience witness a mixed reality experience. People attending the conference from living rooms and home offices around the world could experience the show as avatars watching events unfold in a shared holographic world.

It was made possible with the company’s new mixed-reality platform, Mesh, powered by Azure that allows people in different physical locations to join collaborative and shared holographic experiences on many kinds of devices, Microsoft stated on the company blog.

“This has been the dream for mixed reality, the idea from the very beginning,” said Microsoft Technical Fellow Alex Kipman. “You can actually feel like you’re in the same place with someone sharing content or you can teleport from different mixed reality devices and be present with people even when you’re not physically together.”

Kipman was on the virtual stage as a fully realized holoportation of himself, narrating the show’s opening experience in real time as rays of light that simulated his physical body.

With Mesh users can have collaborative meetings, conduct virtual design sessions, assist others, learn together and host virtual social meetups. People will initially be able to express themselves as avatars in these shared virtual experiences and over time use holoportation to project themselves as their most lifelike, photorealistic selves, the company said.

The new platform is the result of years of Microsoft research and development in areas ranging from hand and eye tracking and HoloLens development to creating persistent holograms and artificial intelligence models that can create expressive avatars. Mesh is built on Azure as it provides security, privacy, data, AI and mixed reality services.

Scenarios where Mesh would be a game changer

Consider a factory floor under construction: Engineers can walk through a holographic model of it checking the equipment in 3D.

Engineering or medical students learning about electric car engines or human anatomy could gather as avatars around a holographic model and remove parts of the engine or peel back muscles to see what’s underneath.

OceanXplorer is a deep sea exploration vessels ever built, with scientists on board to learn from new data constantly collected by instruments and cameras on its deep sea vehicles. It is used to learn more about coral reefs, brine pools, deep hydrothermal vents and minerals around underwater volcanoes, etc.

At Ignite, OceanX announced a new collaboration with Microsoft to create a Mesh-enabled “holographic laboratory” on the ship that scientists could gather in — either in person or virtually from labs and offices around the world — to see 3D holograms of the areas the vehicles are exploring.

“The idea is to take all this amazing scientific data we’re collecting and bring it into a holographic setting and use it as a way to guide scientific missions in real time,” said Vincent Pieribone, vice chairman of OceanX.

The goal is to allow any researcher with a HoloLens 2 or other compatible device, using Microsoft Mesh, to appear around a table as an avatar and point to a particular area on the holographic seafloor that they might have a question about and converse in real time with other scientists about what they are seeing.