Image-based laparoscopic tool detection and tracking using convolutional neural networks: a review of the literature

Abstract

Intraoperative detection and tracking of minimally invasive instruments is a prerequisite for computer- and robotic-assisted surgery. Since additional hardware, such as tracking systems or the robot encoders, are cumbersome and lack accuracy, surgical vision is evolving as a promising technique to detect and track the instruments using only endoscopic images. The present paper presents a review of the literature regarding image-based laparoscopic tool detection and tracking using convolutional neural networks (CNNs) and consists of four primary parts: (1) fundamentals of CNN; (2) public datasets; (3) CNN-based methods for the detection and tracking of laparoscopic instruments; and (4) discussion and conclusion. To help researchers quickly understand the various existing CNN-based algorithms, some basic information and a quantitative estimation of several performances are analyzed and compared from the perspective of ‘partial CNN approaches’ and ‘full CNN approaches’. Moreover, we highlight the challenges related to research of CNN-based detection algorithms and provide possible future developmental directions.

Keywords:

• Tool detection
• tool tracking
• convolutional neural network
• laparoscopic surgery

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the National Key Research and Development Program of China under Grant 2019YFB1311300.