Towards video-based surgical workflow understanding in open orthopaedic surgery

ABSTRACT

Safe and efficient surgical training and workflow management play a critical role in clinical competency and ultimately, patient outcomes. Video data in minimally invasive surgery (MIS) have enabled opportunities for vision-based artificial intelligence (AI) systems to improve surgical skills training and assurance through post-operative video analysis and development of real-time computer-assisted interventions (CAI). Despite the availability of mounted cameras for the operating room (OR), similar capabilities are much more complex to develop for recording open surgery procedures, which has resulted in a shortage of exemplar video-based training materials. In this paper, we present a potential solution to record open surgical procedures using head-mounted cameras. Recorded videos were anonymised to remove patient and staff identifiable information using a machine learning algorithm that achieves state-of-the-art results on the OR Face dataset. We then propose a CNN-LSTM-based model to automatically segment videos into different surgical phases, which has never been previously demonstrated in open procedures. The redacted videos, along with the automatically predicted phases, are then available for surgeons and their teams for post-operative review and analysis. To our knowledge, this is the first demonstration of the feasibility of deploying camera recording systems and developing machine learning-based workflow analysis solutions for open surgery, particularly in orthopaedics.

KEYWORDS:

• Surgical workflow analysis
• surgical data science
• open surgery
• orthopaedics
• machine learning

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1. It is worth mentioning that as the patient is draped, the likelihood of capturing the patient’s face is minimal.

Additional information

Notes on contributors

Abdolrahim Kadkhodamohammadi

Abdolrahim Kadkhodamohammadi, PhD. Lead Computer Vision Engineer at Digital Surgery a Medtronic company, where he develops computer vision solutions to perceive operating rooms. He completed his PhD in computer vision at the University of Strasbourg, France, supervised by Prof. Padoy. He continued there as a postdoc researcher. Previously, he has also worked as a researcher at the Max Planck Institute for Informatics, Saarland University, Germany. He has published and reviewed papers at various international conferences and journals. He has more than 7 years of experience in machine learning and computer vision.LinkedIn: https://www.linkedin.com/in/rkmohammadi/

Nachappa Sivanesan Uthraraj

Nachappa Sivanesan Uthraraj MS (Orthopaedics) and MRCS., (Edinburgh) is a Clinical Research Fellow in Knee Surgery at the University College of London Hospital (UCLH). He graduated with an MBBS degree from the Sri Ramachandra University in Chennai, India. He then worked in various hospitals before pursuing a 3 year training program in basic surgery and Orthopaedic surgery leading to a masters degree in Orthopaedics, by the Rajiv Gandhi University of Health Sciences, in India. He then was inducted as a member into the Royal College of Surgeons in Edinburgh, on successfully passing the required examinations. He has previously worked in clinical, teaching and administrative positions in various hospitals in India. He has also worked as speciality registrar (non-training) at the North Middlesex University Hospital in London prior to his appointment in the current job as a clinical research fellow. He is currently the clinical co-ordinator and research fellow for the “Intelligent Operating Room for Orthopaedic Surgery” project by Digital Surgery, a Medtronic company. He also works on other clinical research projects at the UCLH under Mr Oussedik. Linkedin : http://www.linkedin.com/in/nachappa-sivanesan-uthraraj

Petros Giataganas

Dr Petros Giataganas, PhD. He is the Director of Surgical Technology Research at Medtronic. Prior to joining Medtronic, Petros graduated from the Electrical and Computer Engineering department at the University of Patras, Greece, specializing in robotics and automation, completed his MRes in Medical Robotics and Image Guided Intervention and his PhD in Medical Robotics at Imperial College London, UK. He has over 10 years’ experience in surgical robotics, mechatronics, rapid prototyping and user interfaces along with 3 years’ experience in Medical Devices regulatory compliance. Within Medtronic, Petros is leading the research activities and product development of novel intraoperative hardware solutions. LinkedIn: https://www.linkedin.com/in/petrosgiataganas/

Gauthier Gras

Dr Gauthier Gras, PhD. Lead Systems Engineer, has over 10 years' experience in surgical robotics, software development and systems engineering. He graduated from the Electrical Engineering department of Ecole Normale Supérieure de Cachan, after passing the highly competitive Agrégation nationwide exam. After completing an MRes in Medical Robotics and Image Guided Intervention in Imperial College he continued on to do a PhD there focusing on designing software systems for surgical robotics and probabilistic machine learning methods. Now at Medtronic, he leads the software team developing novel solutions for intraoperative systems. LinkedIn: https://www.linkedin.com/in/gauthier-gras/

Karen Kerr

Dr Karen Kerr, completed her undergraduate (BSc Hons Zoology; 2001) and PhD in infectious diseases (2005) at the University of Edinburgh, UK. She has over 14 years of research management and operations in cancer, surgery, and healthcare technologies. Prior to joining Digital Surgery in 2018 (acquired by Medtronic in February 2020), Karen contributed to generating >£30M grant income at Imperial College London in the translational healthcare technology field. Karen is currently Head of Research Strategy and Operations where she oversees R&D projects, programmes and strategic initiatives. She is responsible for identifying, setting up and managing new initiatives, grants (including two recent Innovate UK grants worth £1M each), and collaborations with international academic and clinical partners. Karen works closely with stakeholders from government, academia, NHS, research networks and industry

Imanol Luengo

Dr Imanol Luengo, graduated in Computer Science (BSc+MSc) and MRes in AI and Computational Optimization from the University of Basque Country. He obtained a PhD in Biomedical Image analysis from the University of Nottingham. He has over 8 years of experience in applied machine learning and computer vision techniques, with focus in the fields of biology, medicine and particularly in surgery. Imanol is currently the Director of Surgical Intelligence at Medtronic, where he leads an AI team to solve challenging vision problems in surgery in order to build new AI-powered products and systems. LinkedIn: https://www.linkedin.com/in/imanol-luengo-84390866/

Sam Oussedik

Mr Sam Oussedik, Consultant Trauma and Orthopaedic Surgeon and Clinical Director of Trauma and Orthopedics at University College Hospital London (UCLH). Dr Oussedik specialises in knee surgery for both sports injuries and degenerative conditions, as well as all forms of trauma surgery. Mr Oussedik studied medicine at St Bartholomew's Medical College, London, and went on to postgraduate training in the field of knee surgery. Mr Oussedik is the Director of Surgical Education at UCLH as well as an active researcher, working to improve results following joint replacement surgery and investigate the role of robotic surgery in orthopedics. Mr Oussedik has been specialty editor for knee surgery for the Bone and Joint Journal, the pre-eminent British Orthopaedic Journal, for the past 5 years and is on the Editorial Board of the American Journal of Sports Medicine, the highest impact global sports surgery journal. He has also co-edited 2 books on knee replacement and has just published a new book on osteotomy surgery for knee conditions.

Danail Stoyanov

Professor Dan Stoyanov, Professor of Robot Vision in the Department of Computer Science at University College London, Director of the Wellcome / EPSRC Centre for Interventional and Surgical Science (WEISS) and a Royal Academy of Engineering Chair in Emerging Technologies.  He is Chief Scientist at Digital Surgery Ltd. Dan first studied electronics and computer systems engineering at King's College London before completing a PhD in Computer Science at Imperial College London where he specialized in medical image computing.  He works on vision problems in minimally invasive surgery especially related to non-rigid structure from motion, scene flow and photometric and geometric camera calibration. His work is applied towards developing image guidance, computational biophotonic imaging modalities and quantitative measurements during robotic assisted minimally invasive procedures. He has over 18 years experience in surgical vision and computational imaging, surgical robotics, image-guided therapies and surgical process analysis.