Towards markerless computer-aided surgery combining deep segmentation and geometric pose estimation: application in total knee arthroplasty

ABSTRACT

Total knee arthroplasty (TKA) is a surgical procedure performed in patients suffering from knee arthritis. The correct positioning of the implants is strongly related to multiple surgical variables that have a tremendous impact on the success of the surgery. Computer-based navigation systems have been investigated and developed in order to assist the surgeon in accurately controlling those surgical variables. The existing technologies are very costly, require additional bone incisions for fixing markers to be tracked, and these markers are usually bulky, interfering with the standard surgical flow. This work presents a markerless navigation system that supports the surgeon in accurately performing the TKA procedure. The proposed system uses a mobile RGB-D camera for replacing the existing optical tracking systems and does not require markers to be tracked. We combine an effective deep learning-based approach for accurately segmenting the bone surface with a robust geometry-based algorithm for registering the bones with pre-operative models. The favourable performance of our pipeline is achieved by (1) employing a semi-supervised labelling approach for generating training data from real TKA surgery data, (2) using effective data augmentation techniques for improving the generalisation capability and (3) using appropriate depth data cleaning strategies. The construction of this complete markerless registration prototype that generalises for unseen intra-operative data is non-obvious, and relevant insights and future research directions can be derived. The experimental results show encouraging performance for video-based TKA.

KEYWORDS:

• Markerless navigation
• deep learning
• image segmentation
• pose estimation
• knee surgery

Acknowledgments

The authors thank the Portuguese Science Foundation and COMPETE2020 program for generous funding through project VisArthro (ref.: PTDC/EEIAUT/3024/2014).

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1. Video education platform for doctors (www.vumedi.com).

Additional information

Funding

This work was supported by the Portuguese Science Foundation and COMPETE2020 program [PTDC/EEIAUT/3024/2014].

Notes on contributors

Inês Félix

Inês Félix recently finished, in 2020, the MSc degree in Biomedical Engineering, with specialization in the field of Clinical Informatics and Bioinformatics. Her interest in Machine Learning and Deep Learning expanded during the development of her master's thesis, which focused on using Deep Learning for image segmentation in medical applications.

Carolina Raposo

Carolina Raposo received the PhD degree from the University of Coimbra, Portugal, in 2017. She is currently a Researcher both at Perceive3D, working in the domain of image-guided surgery, and at the Institute of Systems and Robotics of the University of Coimbra. Her main research interests lie on geometric computer vision, 3D reconstruction and Structure-from-Motion.

Michel Antunes

Michel Antunes received the MSc degree in Biomedical Engineering and the PhD degree in Electrical Engineering from the University of Coimbra in 2008 and 2014, respectively. He has worked in several academic and industrial research institutions, such as the Mitsubishi Electrical Research Laboratories (MERL) in Boston and the Interdisciplinary Centre for Security, Reliability and Trust (SnT) in Luxemburg. He has 10+ years of experience in academic research and in research applied to industrial and medical areas. He has extensive knowledge of computer vision, image processing and machine learning with 30+ peer-reviewed articles and patents. Since 2017 he is a senior R&D engineer at Perceive3D.

Pedro Rodrigues

Pedro Rodrigues received the MSc degree in Biomedical Engineering with a minor in Computer Science in 2010 and the PhD in Electrical Engineering in 2020, both from the University of Coimbra. He is currently working in machine learning research and development.

João P. Barreto

João P. Barreto holds a PhD degree from the University of Coimbra (UC). He was visiting scholar in INRIA Rhone-Alpes, Grenoble, France, and a postdoctoral researcher in the University of Pennsylvania, Philadelphia, before joining the UC as a Professor. João is an acknowledged expert in camera modeling and multi-view 3D reconstruction, being the author of more than 80 peer-reviewed articles in the most prestigious journals and conferences. João is also an entrepreneur having co-founded Perceive3D SA (P3D) in 2013. P3D builds in advanced knowledge in computer vision to provide advanced systems for improving visualization and guiding the surgeon during minimally invasive orthopedic procedures. João has been the CEO of P3D since foundation being responsible for R&D, Business Development and Investor Relations.