A collaborative telerobotics network framework with hand gesture interface and conflict prevention

Abstract

Hand gesture control is an efficient modality for telerobot control because of gesture expressiveness and naturalness. This paper discusses a collaborative cybernetic system, where telerobots are controlled simultaneously by a group of distributed operators over the network to accomplish a task. Computer vision algorithms for hand gesture recognition are introduced to facilitate the human–robot interface. The gestures are converted into commands that are delivered to robots for dexterous task completion. Commands from multiple operators are aggregated by a collaboration protocol into a single control stream. The aggregation is updated according to operators’ performance. A distributed conflict and error detection–prediction network is designed and applied to a case study of collaborative control for a robotic nuclear decommissioning task. Operators use hand gestures to command telerobots to disassemble facilities in a contaminated area. The hypothesis is tested that collaborative control is more effective and less susceptible to conflicts/errors than the standard single-operator control. During collaboration, operators performed gesture commands simultaneously to control a set of robots. The system can reliably recognise operators’ hands with a 96% accuracy in cluttered backgrounds. Collaboration between expert and novice operators can reduce the time to complete a multi-step task by 45% on average.

Keywords:

• collaborative intelligence
• collaborative telerobotics
• computer vision
• conflict and error prevention
• gesture recognition

Acknowledgements

This project was partially supported by the Intelligent Systems and Assistive Technologies (ISAT) Lab and PRISM Center (Production, Robotics and Integrated Software for Manufacturing and Management) at Purdue University. We thank Dr Hoosang Ko for his helpful advice.