![Sample our Mathematics & Statistics journals, sign in here to start your FREE access for 14 days]({"type":"image" , "src" : "/sda/5943/TOC-Subject-Sample-2021-Mathematics-Statistics.jpg"})
ABSTRACT
Temporally locating and classifying instruments in surgical video is useful for the analysis and comparison of surgical techniques. This paper aims to apply action segmentation techniques to temporally segment and classify surgical instruments, and to highlight the utility of this modelling approach through example applications. This paper shows that the action segmentation transformer (ASFormer) architecture with an EfficientNetV2 featurizer performs significantly better in mean average precision than any previous approaches to this task on the Cholec80 dataset. The ASFormer also outperforms Long Short-Term Memory (LSTM) and Multi-Stage Temporal Convolutional Network (MS-TCN) architectures with the same featurizer. This model reduces the need for costly human labelling of surgical video, driving the development of indexed surgical video libraries and instrument usage tracking applications. Examples of these applications are included after the results.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Additional information
Notes on contributors
Bokai Zhang
Bokai Zhang received his MSc degree in Electrical and Computer Engineering at Georgia Institute of Technology in 2018. He is currently working at Johnson & Johnson as a Staff Computer Vision Engineer. He focuses on solving computer vision problems in surgery, such as surgical workflow recognition and surgical instrument recognition. His research interests and expertise are in image classification, object detection, image segmentation, video action recognition, and video action segmentation.
Darrick Sturgeon
Darrick Sturgeon is a data scientist researching surgical technologies at Johnson & Johnson. He received his Master’s degree in Computer Science from Oregon Health and Science University in 2019. His topics of research and publication have included deep learning, computer vision, surgical video analytics, neuroimaging, and functional connectivity.
Arjun Ravi Shankar
Arjun Ravi Shankar received his MSE and BSE Degree in Biomedical Engineering at the University of Pennsylvania. He started working in AI/ML for healthcare at Merck in 2019 and then joined the Johnson & Johnson R&D Leadership Development Program. He is currently working at J&J Digital Solutions as a Deep Learning Data Scientist. His focus is on developing computer vision models to better understand surgical procedures and improve surgical training.
Varun Kejriwal Goel
Varun Kejriwal Goel is a physician with a background in neuroscience, currently training in general surgery at UCSF – East Bay. He worked as a Senior Research Fellow with the R&D arm of Johnson & Johnson MedTech, published research, and worked as the clinical lead for several artificial intelligence models. He has particular experience in computer vision. His broader goal is to encourage cross-sector collaboration and create healthcare technologies that facilitate equitable access to outstanding clinical education and healthcare. He transitions into psychiatric residency at Cornell University, where he will continue to pursue this goal through the lens of mental health.
Jocelyn Barker
Jocelyn Barker received her Ph.D. in BioPhysics at Stanford University. She has worked in data science for six years and is currently the AI/ML Modeling Lead at Johnson & Johnson Digital Solutions. She focuses on developing AI models to better quantify and understand surgical procedures and provide educational information to surgeons. Her work uses deep learning methods in computer vision and video processing combined with clinical factors and outcomes.
Amer Ghanem
Amer Ghanem is a director of Machine Learning with the Digital Solutions division at Johnson & Johnson MedTech where he leads a group of scientists and engineers that are developing Machine Learning models and AI infrastructure. Amer’s background is in Computer Science and he holds a Ph.D. in Computer Science and Engineering from the University of Cincinnati.
Philip Lee
Philip Lee received his MSE in Biomedical Engineering from the University of Michigan in 2018. He previously worked in the additive manufacturing space, utilizing 3D modeling to virtually simulate craniomaxillofacial procedures and design patient-specific guides and implants. He is currently working at Johnson & Johnson MedTech as a Clinical Engineer for Digital Solutions, where he focuses on the development of AI-powered platforms meant to transform operating rooms.
Meghan Milecky
Meghan Milecky received her BS in Biomedical Sciences from Texas A&M University and currently is a senior clinical lead at Johnson & Johnson MedTech. One of her main goals at Johnson & Johnson is transforming clinical data for better patient care and ongoing clinical research. She works closely with leading surgeons across the US and Europe to develop surgical AI metrics and data insights through surgical video analysis.
Natalie Stottler
Natalie Stottler received her MSc degree in Mechanical Engineering: Biomechanics from Stanford University. She currently works at Johnson & Johnson as a Senior Product Manager covering the surgical learning space, with a particular interest in the application of AI to surgeon training.
Svetlana Petculescu
Svetlana Petculescu is a director of Product Management with the Digital Solutions division at Johnson & Johnson MedTech where she leads a group of product managers that are developing cloud/AI-based applications in surgical learning and medical 3D imaging.