The AI-Medic: an artificial intelligent mentor for trauma surgery

ABSTRACT

Telementoring generalist surgeons as they treat patients can be essential when in situ expertise is not available. However, unreliable network conditions, poor infrastructure, and lack of remote mentors availability can significantly hinder remote intervention. To guide medical practitioners when mentors are unavailable, we present the AI-Medic, the initial steps towards an intelligent artificial system for autonomous medical mentoring. A Deep Learning model is used to predict medical instructions from images of surgical procedures. An encoder-decoder model was trained to predict medical instructions given a view of a surgery. The training was done using the Dataset for AI Surgical Instruction (DAISI), a dataset including images and instructions providing step-by-step demonstrations of 290 different surgical procedures from 20 medical disciplines. The predicted instructions were evaluated using cumulative BLEU scores and input from expert physicians. The evaluation was performed under two settings: with and without providing the model with prior information from test set procedures. According to the BLEU scores, the predicted and ground truth instructions were as high as $86\pm 1$% similar. Additionally, expert physicians subjectively assessed the algorithm subjetively and considered that the predicted descriptions were related to the images. This work provides a baseline for AI algorithms assisting in autonomous medical mentoring.

KEYWORDS:

• Datasets
• neural networks
• telementoring
• surgery

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the Office of the Assistant Secretary of Defense for Health Affairs under Award No. W81XWH-14-1-0042. Opinions, interpretations, conclusions and recommendations are those of the author and are not necessarily endorsed by the funders.

Notes on contributors

Edgar Rojas-Muñoz

Dr. Edgar Rojas-Muñoz completed his doctoral studies in the School of Industrial Engineering at Purdue University. He was member of the Intelligent Systems and Assistive Technologies (ISAT) Laboratory at Purdue University. He completed his Licenciatura in Computer Engineering from the Instituto Tecnológico de Costa Rica. His research interests are on gesture understanding, semiotics, human-computer interaction and augmented reality.

Kyle Couperus

Dr. Kyle Couperus, MD is a board certified emergency medicine physician in Tacoma, Washington. He is affiliated with the Emergency Medicine Department of the Madigan Army Medical Center. He completed this residency in the Madigan Healthcare System, and his medical school in the School of Medicina and Biomedical Sciences of the University at Buffalo. He is also a certified member of the American Board of Emergency Medicine.

Juan P. Wachs

Dr. Juan Wachs is an Associate Professor in the School of Industrial Engineering at Purdue University and Adjunct Associate Professor of Surgery at IU School of Medicine. He is the director of the Intelligent Systems and Assistive Technologies (ISAT) Lab at Purdue, and he is affiliated with the Regenstrief Center for Healthcare Engineering. He completed postdoctoral training at the Naval Postgraduate School’s MOVES Institute under a National Research Council Fellowship from the National Academies of Sciences. Dr. Wachs received his B.Ed.Tech in Electrical Education in ORT Academic College, at the Hebrew University of Jerusalem campus. His M.Sc and Ph.D in Industrial Engineering and Management from the Ben-Gurion University of the Negev, Israel.