AVICol: Adaptive Visual Instruction for Remote Collaboration Using Mixed Reality

Abstract

This article describes a mixed reality visual instruction approach for remote collaboration between a trainee and an expert. The expert authors the visual instructions through a virtual reality interface. The instructions are shown to the trainee overlaid onto the workspace using an augmented reality interface. The approach achieves effectiveness and efficiency by addressing three challenges. First, the expert-authored visual instructions are shown to the trainee by taking into account occlusions with the 3D workspace; Second, in addition to abstract visual instructions implemented by arrows, the expert can also author highly suggestive instructions by depicting the target state of the workspace realistically by selecting, copying, pasting, and repositioning workspace objects; Third, multiple instructions can be concatenated in sequences that the trainee executes on their own, without any additional guidance from the expert; The approach has been evaluated in a controlled user study with three experiments. The experiment verification confirms that compared to the conventional instruction, this approach achieves significantly lower error rates, shorter task completion times, and lower rotation angular errors. Moreover, the approach allows the trainee to execute the entire sequence robustly, without real-time instruction from the expert.

Keywords:

• Object manipulation
• remote collaboration
• synchronous collaboration
• asynchronous collaboration
• trainee expert collaboration
• mixed reality
• augmented reality
• virtual reality

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 Microsoft HoloLens 2. https://www.microsoft.com/en-us/hololens.

2 Vive Cosmos. https://www.vive.com/cn/product/vive-cosmos/overview.

3 Unity 2020.3.20f. https://unity3d.com.

4 Mirror. https://mirror-networking.com/.

Additional information

Notes on contributors

Lili Wang

Lili Wang received her PhD degree from Beihang University, Beijing, China. She is a professor with the School of Computer Science and Engineering of Beihang University and a researcher with the State Key Laboratory of Virtual Reality Technology and Systems. Her interests include virtual reality, real-time rendering and HCI.

Xiangyu Li

Xiangyu Li is a master student in the School of Computer Science and Engineering of Beihang University, China. His current research focuses on virtual reality, augmented reality, and HCI.

Jian Wu

Jian Wu is a PhD student in the School of Computer Science and Engineering of Beihang University, China. His current research focuses on virtual reality, augmented reality, visualization and HCI.

Dong Zhou

Zhou Dong is a full professor at Beihang University. He received his PhD in System Engineering at Beihang University. His research interests are multidisciplinary, including virtual and augmented reality, computer-aided design, industrial maintenance and assembly, human factor and knowledge management.

Im Sio Kei

Im Sio Kei received his PhD degree from Queen Mary University of London, United Kingdom. He is a professor at the Faculty of Applied Sciences, Macau Polytechnic University. His research interests include video coding, image processing, machine learning for NLP and multimedia.

Voicu Popescu

Voicu Popescu received his PhD degree in computer science from the University of North Carolina at Chapel Hill, USA in 2001. He is an associate professor with the Computer Science Department of Purdue University. His research interests lie in the areas of computer graphics, computer vision, and visualization.