Surgical instrument recognition for instrument usage documentation and surgical video library indexing

References

• Alshirbaji TA, Jalal NA, Docherty PD, Neumuth T, Möller K. 2021. A deep learning spatial-temporal framework for detecting surgical tools in laparoscopic videos. Biomed Signal Process Control. 68:102801. doi:10.1016/j.bspc.2021.102801.
   Google Scholar
• Alshirbaji TA, Jalal NA, Möller K. 2018. Surgical tool classification in laparoscopic videos using convolutional neural network. Curr Directions Biomed Eng. 4(1):407–410. doi:10.1515/cdbme-2018-0097.
   Google Scholar
• Alshirbaji TA, Jalal NA, Möller K. 2020. A convolutional neural network with a two-stage lstm model for tool presence detection in laparoscopic videos. Curr Directions Biomed Eng. 6(1). doi:10.1515/cdbme-2020-0002.
   Google Scholar
• Arnab A, Dehghani M, Heigold G, Sun C, Lučić M, Schmid C 2021. Vivit: a video vision transformer. Proceedings of the IEEE/CVF International Conference on Computer Vision. p. 6836–6846. https://iccv2021.thecvf.com/home
   Google Scholar
• Ban Y, Rosman G, Eckhoff JA, Ward TM, Hashimoto DA, Kondo T, Iwaki H, Meireles OR, Rus D. 2022. Supr-gan: surgical prediction gan for event anticipation in laparoscopic and robotic surgery. IEEE Rob Autom Lett. 7(2):5741–5748. doi:10.1109/LRA.2022.3156856.
   Google Scholar
• Bouget D, Benenson R, Omran M, Riffaud L, Schiele B, Jannin P. 2015. Detecting surgical tools by modelling local appearance and global shape. IEEE Trans Med Imaging. 34(12):2603–2617. doi:10.1109/TMI.2015.2450831.
   Google Scholar
• Chen W, Feng J, Lu J, Zhou J. 2018. Endo3d: online workflow analysis for endoscopic surgeries based on 3d cnn and lstm. In: oR 2.0 Context-Aware operating theaters, computer assisted robotic endoscopy, clinical image-based procedures, and skin image analysis. Cham: Springer; p. 97–107.
   Google Scholar
• Colleoni E, Moccia S, Du X, De Momi E, Stoyanov D. 2019. Deep learning based robotic tool detection and articulation estimation with spatio-temporal layers. IEEE Rob Autom Lett. 4(3):2714–2721. doi:10.1109/LRA.2019.2917163.
   Google Scholar
• Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, et al. 2020. An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:201011929.
   Google Scholar
• Farha YA, Gall J 2019. Ms-tcn: multi-stage temporal convolutional network for action segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; Long Beach, CA, USA. p. 3575–3584.
   Google Scholar
• Feldman LS, Pryor AD, Gardner AK, Dunkin BJ, Schultz L, Awad MM, Ritter EM. 2020. Sages video-based assessment (vba) program: a vision for life-long learning for surgeons. Surg Endosc. 34(8):3285–3288. doi:10.1007/s00464-020-07628-y.
   Google Scholar
• Hochreiter S, Schmidhuber J. 1997. Long short-term memory. Neural Comput. 9(8):1735–1780. doi:10.1162/neco.1997.9.8.1735.
   Google Scholar
• Jin Y, Dou Q, Chen H, Yu L, Qin J, Fu CW, Heng PA. 2017. Sv-rcnet: workflow recognition from surgical videos using recurrent convolutional network. IEEE Trans Med Imaging. 37(5):1114–1126. doi:10.1109/TMI.2017.2787657.
   Google Scholar
• Jin Y, Li H, Dou Q, Chen H, Qin J, Fu CW, Heng PA. 2020. Multi-task recurrent convolutional network with correlation loss for surgical video analysis. Med Image Anal. 59:101572. doi:10.1016/j.media.2019.101572.
   Google Scholar
• Jin Y, Long Y, Chen C, Zhao Z, Dou Q, Heng PA. 2021. Temporal memory relation network for workflow recognition from surgical video. IEEE Trans Med Imaging. 40(7):1911–1923. doi:10.1109/TMI.2021.3069471.
   Google Scholar
• Kabataş SN, Sarkayaş D 2021. Surgical activity recognition with transformer networks. 2021 29th Signal Processing and Communications Applications Conference (SIU); Istanbul, Turkey. IEEE. p. 1–4.
   Google Scholar
• Kadkhodamohammadi A, Luengo I, Stoyanov D. 2022. Patg: position-aware temporal graph networks for surgical phase recognition on laparoscopic videos. Int J Comput Assist Radiol Surg. 17(5):849–856. doi:10.1007/s11548-022-02600-8.
   Google Scholar
• Kondo S. 2021. Lapformer: surgical tool detection in laparoscopic surgical video using transformer architecture. Comput Methods Biomech Biomed Eng: Imaging & Visualization. 9(3):302–307. doi:10.1080/21681163.2020.1835550.
   Google Scholar
• Li SJ, AbuFarha Y, Liu Y, Cheng MM, Gall J 2020. Ms-tcn++: multi-stage temporal convolutional network for action segmentation. p. 1–1. doi:10.1109/TPAMI.2020.3021756.
   Google Scholar
• Liu D, Wei Y, Jiang T, Wang Y, Miao R, Shan F, Li Z 2020. Unsupervised surgical instrument segmentation via anchor generation and semantic diffusion. International Conference on Medical Image Computing and Computer-Assisted Intervention; Lima, Peru. Springer. p. 657–667.
   Google Scholar
• Long Y, Wu JY, Lu B, Jin Y, Unberath M, Liu YH, Heng PA, Dou Q 2021. Relational graph learning on visual and kinematics embeddings for accurate gesture recognition in robotic surgery. 2021 IEEE International Conference on Robotics and Automation (ICRA); Xi'an China. IEEE. p. 13346–13353.
   Google Scholar
• Nwoye CI, Alapatt D, Yu T, Vardazaryan A, Xia F, Zhao Z, Xia T, Jia F, Yang Y, Wang H, et al. 2022. Cholectriplet2021: a benchmark challenge for surgical action triplet recognition. arXiv preprint arXiv:220404746.
   Google Scholar
• Qin Y, Pedram SA, Feyzabadi S, Allan M, McLeod AJ, Burdick JW, Azizian M 2020. Temporal segmentation of surgical sub-tasks through deep learning with multiple data sources. 2020 IEEE International Conference on Robotics and Automation (ICRA); Paris, France. IEEE. p. 371–377.
   Google Scholar
• Ramesh S, Dall’Alba D, Gonzalez C, Yu T, Mascagni P, Mutter D, Marescaux J, Fiorini P, Padoy N. 2021. Multi-task temporal convolutional networks for joint recognition of surgical phases and steps in gastric bypass procedures. Int J Comput Assist Radiol Surg. 16(7):1111–1119. doi:10.1007/s11548-021-02388-z.
   Google Scholar
• Sharghi A, He Z, Mohareri O. 2022. Activity detection in long surgical videos using spatio-temporal models. arXiv preprint arXiv:220502805.
   Google Scholar
• Tan M, Le Q 2021. Efficientnetv2: smaller models and faster training. International Conference on Machine Learning. PMLR. p. 10096–10106.
   Google Scholar
• Twinanda AP, Shehata S, Mutter D, Marescaux J, De Mathelin M, Padoy N. 2016. Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans Med Imaging. 36(1):86–97. doi:10.1109/TMI.2016.2593957.
   Google Scholar
• Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Ł K, Polosukhin I. 2017. Attention is all you need. Adv Neural Inf Process Syst. 30:5998-–6008.
   Google Scholar
• Wang Z, Ding X, Zhao W, Li X. 2022. Less is more: surgical phase recognition from timestamp supervision. arXiv preprint arXiv:220208199.
   Google Scholar
• Wei H, Rudzicz F, Fleet D, Grantcharov T, Taati B 2021. Intraoperative adverse event detection in laparoscopic surgery: stabilized multi-stage temporal convolutional network with focal-uncertainty loss. Machine Learning for Healthcare Conference. PMLR. p. 283–307.
   Google Scholar
• Yang Z, Simon R, Linte C. 2022. A weakly supervised learning approach for surgical instrument segmentation from laparoscopic video sequences In: Linte, Cristian A., Siewerdsen, Jeffrey H., editors. Medical Imaging 2022: image-Guided Procedures, Robotic Interventions, and Modeling. Vol. 12034. Bellingham, Washington USA: SPIE; p. 412–417.
   Google Scholar
• Yi F, Wen H, Jiang T 2021. Asformer: transformer for action segmentation. arXiv preprint arXiv:211008568.
   Google Scholar
• Zhang B, Abbing J, Ghanem A, Fer D, Barker J, Abukhalil R, Goel VK, Milletar F. 2022. Towards accurate surgical workflow recognition with convolutional networks and transformers. Comp Met Biomech Biomed Eng: Imaging Visualization. 10:349–356. doi:10.1080/21681163.2021.2002191.
   Google Scholar
• Zhang B, Ghanem A, Simes A, Choi H, Yoo A. 2021a. Surgical workflow recognition with 3dcnn for sleeve gastrectomy. Int J Comput Assist Radiol Surg. 16(11):2029–2036. doi:10.1007/s11548-021-02473-3.
   Google Scholar
• Zhang B, Ghanem A, Simes A, Choi H, Yoo A, Min A. 2021b. Swnet: surgical workflow recognition with deep convolutional network In: Heinrich, Mattias, Dou, Qi, de Bruijne, Marleen, Lellmann, Jan, Schläfer, Alexander, Ernst, Floris, editors. Proceedings of the Fourth Conference on Medical Imaging with Deep Learning. PMLR; p. 855–869.
   Google Scholar
• Zhao Z, Jin Y, Heng PA. 2022. Trasetr: track-to-segment transformer with contrastive query for instance-level instrument segmentation in robotic surgery. 2022 International Conference on Robotics and Automation (ICRA) Philadelphia, PA, USA. p. 11186–11193. doi:10.1109/ICRA46639.2022.9811873.
   Google Scholar
• Zia A, Hung A, Essa I, Jarc A 2018. Surgical activity recognition in robot-assisted radical prostatectomy using deep learning. International Conference on Medical Image Computing and Computer-Assisted Intervention; Granada, Spain. Springer. p. 273–280.
   Google Scholar