SARTRES: a semi-autonomous robot teleoperation environment for surgery

ABSTRACT

Teleoperated surgical robots can provide immediate medical assistance in austere and hostile environments. However, such scenarios are time-sensitive and require high-bandwidth and low-latency communication links that might be unavailable. The system presented in this paper has a standard surgical teleoperation interface, which provides surgeons with an environment in which they are trained. In our semi-autonomous robotic framework, high-level instructions are inferred from the surgeon’s actions and then executed semi-autonomously on the robot. The framework consists of two main modules: (i) Recognition Module – which recognises atomic sub-tasks (i.e., surgemes) performed at the operator end, and (ii) Execution Module – which executes the identified surgemes at the robot end using task contextual information. The peg transfer task was selected for this paper due to its importance in laparoscopic surgical training. The experiments were performed on the DESK surgical dataset to show our framework’s effectiveness using two metrics: user intervention (in the degree of autonomy) and success rate of surgeme execution. We achieved an average accuracy of 91.5% for surgeme recognition and 86% success during surgeme execution. Furthermore, we obtained an average success rate of 53.9% for the overall task, using a model-based approach with a degree of autonomy of 99.33%.

KEYWORDS:

• Teleoperated Robotic Surgery
• surgical activity recognition
• surgical vision and perception

Acknowledgments

This work was supported by both the Office of the Assistant Secretary of Defense for Health Affairs under Award No. W81XWH-18-1-0769 and by the NSF Center for Robots and Sensors for the Human Well-Being under Award No. CNS-1439717. Y.X. and M.R. also announce support from the NSF FMitF program (CCF-1918327) and the CR-II program (IIS-1850243). Computational infrastructure was partially supported by Microsoft AI for Earth program. Opinions, interpretations, conclusions and recommendations are those of the authors and are not necessarily endorsed by the funders.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the Office of the Assistant Secretary of Defense for Health Affairs [W81XWH-18-1-0769]; NSF FMitF program [CCF-1918327], CR-II program [IIS-1850243]; NSF Center for Robots and Sensors for the Human Well-Being [CNS-1439717]. Computational infrastructure was partially supported by Microsoft AI for Earth program.

Notes on contributors

Md Masudur Rahman

Md Masudur Rahman is a Ph.D. student in the Department of Computer Science at Purdue University, advised by Dr. Yexiang Xue. He is broadly interested in Artificial Intelligence, Machine Learning, and Robotics. He works on designing and building intelligent learning agents (e.g., RL), which can make interpretable critical decisions under uncertainty. Masudur completed M.S. from the Department of Computer Science at the University of Virginia in 2018. Before joining the University of Virginia, he worked as a Lecturer in Computer Science and Engineering Department at the BRAC University. He completed B.Sc. in Computer Science and Engineering from Bangladesh University of Engineering and Technology (BUET) in 2013.

Mythra V. Balakuntala

Mythra V. Balakuntala is currently a Ph.D. student in Collaborative robotics lab at Purdue University supervised by Dr. Richard Voyles. He received  Bachelors of  Technology in Mechanical Engineering from Indian Institute of Technology, Madras in 2015. His research focuses on learning tasks from visual human demonstrations and coaching.

Glebys Gonzalez

Glebys Gonzalez is a PhD student at the Intelligent Systems and Assistive Technologies at Purdue University. Ms. Gonzalez earned her Bachelor’s degree in Computer Science at Simon Bolivar University, Venezuela. Her research focuses on robot coaching and human robot interaction, mainly applied to medical assistive technologies.

Mridul Agarwal

Mridul Agarwal is currently a Ph.D. student in Cloud, Learning, and Networking lab at Purdue University supervised by Dr. Vaneet Aggarwal. He received  Bachelors of  Technology in Electrical Engineering from Indian Institute of Technology, Kanpur in 2014. His research focuses on theoretical foundations of machine learning, communication networks, and game theory.

Upinder Kaur

Upinder Kaur is currently a Ph.D. student in Collaborative robotics lab at Purdue University supervised by Dr. Richard Voyles. She received Masters of  Technology in Robotics and Automation Engineering from Indira Gandhi Delhi Technical University for Women, Delhi in 2017. Her research focuses on robust robotic networks integrated with AI.

Vishnunandan L. N. Venkatesh

Vishnunandan L.N. Venkatesh is currently a Ph.D. student in the Collaborative Robotics Lab at Purdue University.  He completed M.S. from the Department of Electrical Engineering, majoring in Robotics and Automation at Rochester Institute of Technology in 2018. He received  Bachelors of  Technology in Electronics and Communication Engineering from Vellore Institute of Technology, Vellore in 2015. His research interests include Human-Robot Interaction, Machine Learning and Robot Manipulation.

Natalia Sanchez-Tamayo

Natalia Sanchez-Tamayo received B.S. degrees in Mechanical Engineering (2016) and Civil Engineering (2017) at the University of the Andes, Colombia. In 2020, she received a M.S degree in Industrial Engineering at Purdue University, Indiana, USA, where she worked with Dr. Juan Wachs in the Intelligent Systems and Assistive Technologies laboratory (ISAT). She is currently a Ph.D student at the Max-Planck Institute for Intelligent Systems, Stuttgart, Germany in the Haptic Intelligence department supervised by Dr. Katherine Kuchenbecker.

Yexiang Xue

Yexiang Xue is an assistant professor at the department of computer science, Purdue University. Prior to coming to Purdue, Dr. Xue received his Masters and Ph.D. degrees from Cornell University. Dr. Xue received his Bachelor of Science degree in 2011 from Peking University, China. Dr. Xue’s research focuses on developing intelligent systems that tightly integrate decision making with machine learning and probabilistic reasoning under uncertainty. Dr. Xue focuses on developing cross-cutting computational methods with applications to a variety of domains, with an emphasis in the new exciting area of computational sustainability and scientific discovery. Dr. Xue received the innovative application award from IAAI, Ph.D. dissertation award from Cornell, and the Seed for Success award from Purdue.

Richard M. Voyles

Richard M. Voyles is the Daniel C. Lewis Professor of the Polytechnic and a professor of robotics and CPS in the School of Engineering Technology, Purdue University.  Prof. Voyles is a manufacturing fellow at the Indiana Manufacturing Institute, Director of the Purdue Robotics Accelerator, and head of the Collaborative Robotics Lab. He holds a BS in electrical engineering from Purdue, an MS in manufacturing systems engineering from Stanford and a PhD in robotics from Carnegie Mellon. He has been a program director at the National Science Foundation and was Assistant Director of Robotics and Cyber-Physical Systems at the White House Office of Science and Technology Policy. Dr. Voyles’ research interests are in the design of robotic mechanisms for hazardous environments, software architectures for self-adaptive systems, and multi-modal sensor systems for contact-intensive tasks. He has been named a University Faculty Scholar and has received the Seed for Success award from Purdue.

Vaneet Aggarwal

Md Masudur Rahman is a Ph.D. student in the Department of Computer Science at Purdue University, advised by Dr. Yexiang Xue. He is broadly interested in Artificial Intelligence, Machine Learning, and Robotics. He works on designing and building intelligent learning agents (e.g., RL), which can make interpretable critical decisions under uncertainty. Masudur completed M.S. from the Department of Computer Science at the University of Virginia in 2018. Before joining the University of Virginia, he worked as a Lecturer in Computer Science and Engineering Department at the BRAC University. He completed B.Sc. in Computer Science and Engineering from Bangladesh University of Engineering and Technology (BUET) in 2013.

Vaneet Aggarwal (S’08 - M’11 - SM’15) received the B.Tech. degree from the Indian Institute of Technology, Kanpur, India in 2005, and the M.A. and Ph.D. degrees in 2007 and 2010, respectively from Princeton University, Princeton, NJ, USA, all in Electrical Engineering. He is currently an Associate Professor at Purdue University, West Lafayette, IN, where he has been since Jan 2015. He was a Senior Member of Technical Staff Research at AT&T Labs-Research, NJ (2010-2014), Adjunct Assistant Professor at Columbia University, NY (2013-2014), and VAJRA Adjunct Professor at IISc Bangalore (2018-2019). His current research interests are in communications and networking, cloud computing, and machine learning. Dr. Aggarwal received Princeton University’s Porter Ogden Jacobus Honorific Fellowship in 2009, the AT&T Vice President Excellence Award in 2012, the AT&T Key Contributor Award in 2013, the AT&T Senior Vice President Excellence Award in 2014, the 2017 Jack Neubauer Memorial Award recognizing the Best Systems Paper published in the IEEE TRANSACTIONS ON VEHICULAR TECHNOLOGY, and the 2018 Infocom Workshop HotPOST Best Paper Award. He is on the Editorial Board of the IEEE TRANSACTIONS ON COMMUNICATIONS, the IEEE TRANSACTIONS ON GREEN COMMUNICATIONS AND NETWORKING, and the IEEE/ACM TRANSACTIONS ON NETWORKING.

Dr. Juan Wachs is a Faculty Scholar Professor in Industrial Engineering School at Purdue University, Professor of Biomedical Engineering (by courtesy) and an Adjunct Associate Professor of Surgery at IU School of Medicine. He is the director of the Intelligent Systems and Assistive Technologies (ISAT) Lab at Purdue, and he is affiliated with the Regenstrief Center for Healthcare Engineering. He completed postdoctoral training at the Naval Postgraduate School’s MOVES Institute under a National Research Council Fellowship from the National Academies of Sciences. Dr. Wachs received his B.Ed.Tech in Electrical Education in ORT Academic College, at the Hebrew University of Jerusalem campus. His M.Sc and Ph.D in Industrial Engineering and Management from the Ben-Gurion University of the Negev, Israel. He is the recipient of the 2013 Air Force Young Investigator Award, and the 2015 Helmsley Senior Scientist Fellow, and 2016 Fulbright U.S. Scholar, the James A. and Sharon M. Tompkins Rising Star Associate Professor, 2017, and an ACM Distinguished Speaker 2018. He is also the Associate Editor of IEEE Transactions in Human-Machine Systems, Frontiers in Robotics and AI.

Juan Wachs

Mythra V. Balakuntala is currently a Ph.D. student in Collaborative robotics lab at Purdue University supervised by Dr. Richard Voyles. He received  Bachelors of  Technology in Mechanical Engineering from Indian Institute of Technology, Madras in 2015. His research focuses on learning tasks from visual human demonstrations and coaching.