Haptic Feedback Helps Me? A VR-SAR Remote Collaborative System with Tangible Interaction

ABSTRACT

Research on Augmented Reality (AR)/Mixed Reality (MR) remote collaboration for physical tasks remains a compelling and dynamic area of study. AR systems have been developed which transmit virtual annotations between remote collaborators, but there has been little research on how haptic feedback can also be shared. In this paper, we present a Virtual Reality (VR)-Spatial Augmented Reality (SAR) remote collaborative system that provides haptic feedback with tangible interaction between a local worker and a remote expert helper. Using this system, we conducted a within-subject user study to compare two interfaces for remote collaboration between a local worker and expert helper, one with mid-air free drawing (MFD) and one with tangible physical drawing (TPD). The results showed that there were no significant differences with respect to performance time and operation errors. However, users felt that the TPD interface supporting passive haptic feedback could significantly improve the remote experts’ user experience in VR. Our research provides useful information on the way for gesture- and gaze-based multimodal interaction supporting haptic feedback in AR/MR remote collaboration on physical tasks.

Acknowledgments

This research was financially sponsored by the civil aircraft special project (MJZ-2017-G73), Dongguan Science and Technology Equipment Project (KZ2018-05), and 111 project (B13044). Besides, we would like to appreciate the anonymous reviewers for their constructive suggestions for enhancing this paper. In particular, Professor Mark Billinghurst carefully checked the English of the early and final version of the paper, and his constructive comments helped the author to improve the paper. We would like to thank Professor Weiping He for science leadership. Additionally, we thank our schoolfellow for their contribution to this research: Dechuan Han for improving the limitations of the prototype, Mengmeng Sun for the experimental data collection, Hao Lv, Shu Han, Shuo Feng, and Yupeng Hu for preparing some supplementary materials. Moreover, we would also like to thank members of Northwestern Polytechnical University for their participation in the experiment.

Notes

1. https://measuringu.com/sus.

Additional information

Notes on contributors

Peng Wang

Peng Wang is a PhD candidate in the School of Mechanical Engineering at Northwestern Polytechnical University, PR China. He obtained a BS (2012)  and MS (2015)  in Mechanical manufacturing and automation from Lan Zhou University of Technology, PR China. Now his research interests cover VR/AR/MR, AR/MR remote collaboration in manufacturing, HCI.

Xiaoliang Bai

Xiaoliang Bai is an associate professor in the School of Mechanical Engineering at Northwestern Polytechnical University, PR China. He obtained his BS, MS and PhD from Northwestern Polytechnical University, PR China in 1996, 1999 and 2005 respectively. His research interests include AR/MR, remote collaboration, HCI and intelligent manufacture, etc.

Mark Billinghurst

Mark Billinghurst is Professor of Human Computer Interaction at the University of South Australia in Adelaide, Australia. He earned a PhD in 2002 from the University of Washington and researches innovative computer interfaces that explore how virtual and real worlds can be merged, publishing over 300 papers in topics such as wearable computing, empathic computing, Augmented Reality and mobile interfaces.

Shusheng Zhang

Shusheng Zhang is a professor in the School of Mechanical Engineering at Northwestern Polytechnical University, PR China. He obtained both of his BS and MS from Northwestern Polytechnical University, PR China in 1980 and 1984, respectively and his PhD from the University of Havre, France in 1991. His research interests include AR/MR, remote collaboration, digital design and manufacturing, 3D CAD model retrieval.

Dechuan Han

Dechuan Han is a graduate student in the School of School of Mechanical Engineering at Northwestern Polytechnical University, PR China. His research interests include AR/MR, remote collaboration, HCI.

Mengmeng Sun

Mengmeng Sun is a graduate student in the School of School of Mechanical Engineering at Northwestern Polytechnical University, PR China. Her research interests include AR/MR, haptic interaction, HCI.

Zhuo Wang

Zhuo Wang is a PhD candidate in the School of Mechanical Engineering at Northwestern Polytechnical University, PR China. His research interests include AR/MR, HCI, engineering information visualision.

Hao Lv

Hao Lv is a graduate student in the School of School of Mechanical Engineering at Northwestern Polytechnical University, PR China. His research interests include AR/MR, HCI, computer vision.

Shu Han

Shu Han is a graduate student in the School of Mechanical Engineering at Northwestern Polytechnical University, PR China. His research interests include AR/MR, computer vision, industrial design.