Towards real-time multiple surgical tool tracking

ABSTRACT

Surgical tool tracking is an essential building block for computer-assisted interventions (CAI) and applications like video summarisation, workflow analysis and surgical navigation. Vision-based instrument tracking in laparoscopic surgical data faces significant challenges such as fast instrument motion, multiple simultaneous instruments and re-initialisation due to out-of-view conditions or instrument occlusions. In this paper, we propose a real-time multiple object tracking framework for whole laparoscopic tools, which extends an existing single object tracker. We introduce a geometric object descriptor, which helps with overlapping bounding box disambiguation, fast motion and optimal assignment between existing trajectories and new hypotheses. We achieve 99.51% and 75.64% average accuracy on ex-vivo robotic data and in-vivo laparoscopic sequences respectively from the Endovis’15 Instrument Tracking Dataset. The proposed geometric descriptor increased the performance on laparoscopic data by 32%, significantly reducing identity switches, false negatives and false positives. Overall, the proposed pipeline can successfully recover trajectories over long-sequences and it runs in real-time at approximately 25–29 fps.

KEYWORDS:

• Multiple tool tracking
• Surgical data science
• Computer-Assisted Interventions

Disclosure statement

No potential conflict of interest was reported by the authors.

Supplemental material

Supplemental data for this article can be accessed here

Notes

1. https://endovissub-instrument.grand-challenge.org/

Additional information

Notes on contributors

Maria Robu

Dr Maria Robu, completed her Ph.D. at the Wellcome / EPSRC Centre for Interventional and Surgical Science (WEISS), UCL, UK where she focused on computer vision solutions for augmented reality systems during laparoscopic interventions. Before that, she graduated with distinctions from UCL in the M.Sc. in Computer Graphics, Vision and Imaging and the M.Res. in Medical Imaging. She is a recipient of the Google Anita Borg Scholarship 2014. Maria now works as a Principal Computer Vision Engineer at Digital Surgery, a Medtronic company. Linkedin: https://www.linkedin.com/in/mariarrobu/

Abdolrahim Kadkhodamohammadi

Dr Abdolrahim Kadkhodamohammadi, PhD. Lead Computer Vision Engineer at Digital Surgery a Medtronic company, where he develops computer vision solutions to perceive operating rooms. He completed his PhD in computer vision at the University of Strasbourg, France, supervised by Prof. Padoy. He continued there as a postdoc researcher. Previously, he has also worked as a researcher at the Max Planck Institute for Informatics, Saarland University, Germany. He has published and reviewed papers at various international conferences and journals. He has more than 7 years of experience in machine learning and computer vision. LinkedIn: https://www.linkedin.com/in/rkmohammadi/

Imanol Luengo

Dr Imanol Luengo, graduated in Computer Science (BSc+MSc) and MRes in AI and Computational Optimization from the University of Basque Country. He obtained a PhD in Biomedical Image analysis from the University of Nottingham. He has over 9 years of experience in applied machine learning and computer vision techniques, with focus in the fields of biology, medicine and particularly in surgery. Imanol is currently the Director of Surgical Intelligence at Medtronic, where he leads an AI team to solve challenging vision problems in surgery in order to build new AI-powered products and systems. LinkedIn: https://www.linkedin.com/in/imanol-luengo-84390866/

Danail Stoyanov

Dan Stoyanov, Professor of Robot Vision in the Department of Computer Science at University College London, Director of the Wellcome / EPSRC Centre for Interventional and Surgical Science (WEISS) and a Royal Academy of Engineering Chair in Emerging Technologies.  He is Chief Scientist at Digital Surgery Ltd. Dan first studied electronics and computer systems engineering at King's College London before completing a PhD in Computer Science at Imperial College London where he specialized in medical image computing.  He works on vision problems in minimally invasive surgery especially related to non-rigid structure from motion, scene flow and photometric and geometric camera calibration. His work is applied towards developing image guidance, computational biophotonic imaging modalities and quantitative measurements during robotic assisted minimally invasive procedures. He has over 18 years experience in surgical vision and computational imaging, surgical robotics, image-guided therapies and surgical process analysis.