Image-based laparoscopic tool detection and tracking using convolutional neural networks: a review of the literature

Figures & data

Figure 1. The architecture of the convolutional neural network.

Figure 2. The seven surgical tools used in the Cholec80 dataset.

Figure 3. Full pipeline of the method based on LSD.

Figure 4. Full pipeline of the method based on the Hierarchical Hidden Markov Model.

Figure 5. Full pipeline of the method based on Random Forests.

Figure 6. Full pipeline of the method based on Model Ensembling.

Figure 7. Full pipeline of the method based on Temporal Smoothing.

Figure 8. Full pipeline of the method based on Optical Flow.

Figure 9. Full pipeline of the method based on a detection-regression network.

Figure 10. Full pipeline of the method based on ‘coarse to fine.’

Figure 11. Region Proposal Network (RPN).

Figure 12. Full pipeline of the method based on a Region Proposal Network.

Figure 13. Full pipeline of the method based on YOLO.

Figure 14. Full pipeline of the method based on LSTM.

Figure 15. Full pipeline of the method based on U-Net.

Figure 16. Full pipeline of the Multi-task Learning Network based on ResNet-50.

Figure 17. Full pipeline of the Multi-task Learning Network based on LSTM Units.

Figure 18. Full pipeline of the method based on an Hourglass Network.

Figure 19. Full pipeline of the method based on Modified ResNet18.

Figure 20. Full pipeline of the method based on ConvLSTM.

Table 1. Summary of the comparison of several CNN-based algorithms.

Dataset	Reference	Type	Detection accuracy (%)	Speed
Cholec80	Twinanda [15]	Partial CNN	81	–
	Vardazaryan [49]	Full CNN	88	–
	Jin [3]	Full CNN	89.1	3.33 fps
	Du [10]	Full CNN	90.68	–
	Nwoye [8]	Full CNN	92.9	–
M2CAI16	Sahu [21]	Partial CNN	61.5	–
	Wang [18]	Partial CNN	63.8	–
	Sahu [17]	Partial CNN	65	48.9 fps
	Choi [28]	Full CNN	72.6	38 fps
	Simonyan [19]	Full CNN	84.7	–
	Al Hajj [32]	Full CNN	97.89	–
Endovis	Laina [39]	Full CNN	92.6	19.23 fps
	Zhao [2]	Full CNN	95	–
	Zhao [44]	Full CNN	100	43.5 fps
	Liu [45]	Full CNN	100	37.0 fps
ATLAS Dione	Sarikaya [11]	Full CNN	91	10 fps
	Zhao [44]	Full CNN	91.6	43.5 fps
	Liu [45]	Full CNN	98.5	37.0 fps

Twinanda AP, Shehata S, Mutter D, et al. EndoNet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans Med Imaging. 2017;36(1):86–97.

 PubMed Web of Science ®Google Scholar

Vardazaryan A, Mutter D, Marescaux J, et al. Weakly-supervised learning for tool localization in laparoscopic videos. In: Stoyanov D, Taylor Z, Balocco S, Sznitman R, editors. Intravascular imaging and computer assisted stenting and large-scale annotation of biomedical data and expert label synthesis. LABELS 2018, CVII 2018, STENT 2018. Lecture notes in computer science. Vol 11043. Cham; Heidelberg: Springer; 2018. p. 169–179.

 Google Scholar

Jin Y, Li H, Dou Q, et al. Multi-task recurrent convolutional network with correlation loss for surgical video analysis. Med Image Anal. 2020;59:1–14.

 Web of Science ®Google Scholar

Du X, Kurmann T, Chang P, et al. Articulated multi-instrument 2-D pose estimation using fully convolutional networks. IEEE Trans Med Imaging. 2018;37(5):1276–1287.

 PubMed Web of Science ®Google Scholar

Nwoye CI, Mutter D, Marescaux J, et al. Weakly supervised convolutional LSTM approach for tool tracking in laparoscopic videos. Int J Comput Assist Radiol Surg. 2019;14(6):1059–1067.

 PubMed Web of Science ®Google Scholar

Sahu M, Mukhopadhyay A, Szengel A, et al. Addressing multi-label imbalance problem of surgical tool detection using CNN. Int J Comput Assist Radiol Surg. 2017;12(6):1013–1020.

 PubMed Web of Science ®Google Scholar

Wang S, Raju A, Huang J. Deep learning based multi-label classification for surgical tool presence detection in laparoscopic videos. Proceedings of 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017). Melbourne (VIC): IEEE; 2017. p. 620–623.

 Google Scholar

Sahu M, Mukhopadhyay A, Szengel A, et al. Tool and Phase recognition using contextual CNN features. arXiv: Computer Vision and Pattern Recognition. 2016.

 Google Scholar

Choi B, Jo K, Choi S, et al. Surgical-tools Detection based on Convolutional Neural Network in Laparoscopic Robot-assisted Surgery. In: Proceedings of Annual International Conference of the IEEE Engineering in Medicine and Biology Society. New York (NY): IEEE; 2017. p. 1756–1759.

 Google Scholar

Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington (DC): IEEE Computer Society; 2014.

 Google Scholar

Al Hajj H, Lamard M, Conze P, et al. Monitoring tool usage in surgery videos using boosted convolutional and recurrent neural networks. Med Image Anal. 2018;47:203–218.

 PubMed Web of Science ®Google Scholar

Laina I, Rieke N, Rupprecht C, et al. Concurrent segmentation and localization for tracking of surgical instruments. In: Descoteaux M, Maier-Hein L, Franz A, Jannin P, Collins DL, Duchesne S, editors. Medical image computing and computer-assisted intervention − MICCAI 2017. MICCAI 2017. Lecture notes in computer Science. Vol 10434. Cham; Heidelberg: Springer; 2017. p. 664–672.

 Google Scholar

Zhao Z, Voros S, Chen Z, et al. Surgical tool tracking based on two CNNs: from coarse to fine. J Engg-Joe. 2019;2019(14):467–472.

 Google Scholar

Zhao Z, Cai T, Chang F, et al. Real-time surgical instrument detection in robot-assisted surgery using a convolutional neural network cascade. Healthc Technol Lett. 2019;6(6):275–279.

 PubMed Web of Science ®Google Scholar

Liu Y, Zhao Z, Chang F, et al. An anchor-free convolutional neural network for real-time surgical tool detection in robot-assisted surgery. IEEE Access. 2020;8:78193–78201.

 Web of Science ®Google Scholar

Sarikaya D, Corso JJ, Guru KA. Detection and localization of robotic tools in robot-assisted surgery videos using deep neural networks for region proposal and detection. IEEE Trans Med Imaging. 2017;36(7):1542–1549.

 PubMed Web of Science ®Google Scholar