Imagine you put on a virtual reality (VR) headset and immediately find yourself in an entirely different real-world place. It’s like finding a warp zone in Super Mario Bros. Or walking through Alice’s magical mirror into Wonderland or the wardrobe door into Narnia. Only, it is not a game or a fantasy novel, but reality!

We can now remotely visit sites in 360° and real-time. Our new open-access paper in Manufacturing Letters details our research collaboration with Stanley Black & Decker and Avatour. We conceptualized, realized, and tested a remote presence technology that is immersive, real-time, and interactive – all simultaneously! This emerging technology represents a radical new way of organizing remote factory meetings, shop-floor tours, inspections, audits, and training.

A problem for Stanley Black & Decker

Stanley Black & Decker (SB&D) is a leading manufacturer of construction tools and related products, operating over 100 factories worldwide. A pain point for SB&D is that setting up machines from one part to another wastes time and resources. One reason is that changeovers must be supervised by certified changeover experts – who are scarce and busy. When these experts are traveling between sites or running between shopfloor areas, machines and operators must wait. If the changeovers could be supervised remotely, SB&D would save traveling budgets and increase the uptime of its production assets. Add the decreasing availability of skilled changeover experts, and the case for remote, real-time changeover supervision is clear.

However, traditional remote meeting technologies are not suitable. The cheapest and most straightforward solution is to take a web conference to the shop floor using a computer or handheld device and web meeting software like Zoom & co. However, this solution provides a very limited view for remote participants. It can be improved by using first-person-view video transmission technology such as RealWare, but that technology prevents the remote viewer from looking freely at the process and context.

A more immersive alternative is virtual reality (VR) with head-mounted displays. Yet VR solutions offered by, for example, Openspace or Holobuilder can only provide a near-real-time experience (see also our use of visiting factories in VR). Finally, recent developments in live-streaming technology allow for the real-time streaming of events to a large number of remote participants, yet rarely with the opportunity for two-way communication.

Our idea was simple: let’s combine the best from all these to enable remote factory meetings!

Toward interactive 360° live-streaming in manufacturing

In a research project with SB&D, ETH students Michael Stegmaier and Cesare Primultini led our field test of interactive mixed reality live streaming technology in manufacturing. Michael and Cesare conceptualized two ways to transmit live-streaming 360° video from the shop floor of SB&D: A first-person view where a camera was mounted to a hard hat and a fixed-position view where the camera could be mounted inside a machine during a changeover. Live-streamed 360° video would be transmitted to a cloud, where participants would meet virtually from anywhere (see figure below). Back in 2021, we identified Avatour, a startup from California, as our project’s only viable software provider.

We tested the technology during a three-hour machine setup at SB&D’s factory in Giessen, Germany, in April 2021. During the field test, I put on my Oculus VR headset in my office at ETH Zurich in Switzerland and immediately “warped” into the SB&D shopfloor in Germany. In the mixed reality environment, I could directly observe the changeover process and communicate with shopfloor personnel and other virtual participants.

This was a real but also surreal experience; it may have been the first time anyone ever immersively and interactively visited a remote factory in 360° in real-time.

What’s next for remote factory meetings in 360°?

At the time we tested this technology, it worked! But it wasn’t ready for prime time in 2021. We experienced drawbacks and issues concerning limited bandwidth at the SB&D site, limitations of the Avatour platform, and cybersickness. An SB&D engineer concluded that the technology was “very good to get an overview of the real production environment,” it provided a “big improvement compared to usual camera streaming.”

However, the experienced limitations did not justify implementation back then. Despite the successful field test, the technology cannot yet give the feeling of physically being on the real shop floor. Yet, given the rapid pace of software and sensor development, the technical drawbacks are expected to be solved soon. You can expect a radically new way of “traveling” to remote sites without traveling.

Where can this technology find good use? High-potential use cases are safety audits, training programs, and tasks that require the presence of an expert or coach. Other use cases are factory tours, external or internal audits, technology transfer projects, inspections, and on-the-job coaching of individuals. But let’s see where this technology takes us: An exciting part of first-time warping in Super Mario Bros. is that you never know where you pop up on the other side…

Read more

Netland, T., Stegmaier, M., Primultini, C., & Maghazei, O. (2023). Interactive mixed reality live streaming technology in manufacturing. Manufacturing Letters. URL: https://www.sciencedirect.com/science/article/pii/S2213846323002018