HAPNet: hierarchically aggregated pyramid network for real-time stereo matching

ABSTRACT

Recovering the 3D shape of the surgical site is crucial for multiple computer-assisted interventions. Stereo endoscopes can be used to compute 3D depth but computational stereo is a challenging, non-convex and inherently discontinuous optimisation problem. In this paper, we propose a deep learning architecture which avoids the explicit construction of a cost volume of similarity which is one of the most computationally costly blocks of stereo algorithms. This makes training our network significantly more efficient and avoids the needs for large memory allocation. Our method performs well, especially around regions comprising multiple discontinuities around surgical instrumentation or around complex small structures and instruments. The method compares well to the state-of-the-art techniques while taking a different methodological angle to computational stereo problem in surgical video.

KEYWORDS:

• Convolutional neural networks
• colonoscopy
• computer-aided diagnosis

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Patrick Brandao

Patrick Brandao, I obtained an integrated Master’s Degree in Biomedical Engineering from the University of Minho, Portugal, in 2014. During the following year, I worked as a research engineer at the University of Coimbra, where I developed methods capable of automatically detecting traffic incidents using real-time highway security videos. I am currently a PhD student at the Centre for Medical Image Computing (CMIC) at UCL. I have a funded position in the Endoo project, which aims at developing an integrated robotic platform for colonoscopy. My research focuses on machine learning algorithms for recognition, motion estimation and localisation mapping within the body using cameras, and potentially new designs for imaging systems.

Dimitris Psychogyios

Evangelos Mazomenos, I am a Senior Research Associate at the Department of Computer Science affiliated with the Wellcome / EPSRC Centre for Interventional and Surgical Sciences (WEISS) and the Centre for Medical Image Computing (CMIC) at University College London. My research focuses on autonomous intelligent systems for optimising surgical interventions, diagnosis, and delivery of health care. I graduated, in 2006, from the University of Patras, Greece, with a Diploma (5-year MEng) in Electrical and Computer Engineering and obtained my PhD in Electronics and Electrical Engineering from the University of Southampton in 2012. For my PhD research, I was awarded the Institute of Engineering and Technology Leslie H. Paddle Scholarship in 2009. My main research investigates data science and computer vision methods for performance evaluation and workflow analysis in image-guided interventions. I am also interested in surgical robotics, medical image analysis, biomedical informatics, and embedded sensing systems.

Danail Stoyanov

Danail Soyanov, I am a Professor of Robot Vision at the Department of Computer Science and Director of the Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS). I am also on the board of the Centre for Medical Image Computing (CMIC) and the UCL Robotics Institute.My research interests and expertise are in surgical vision and computational imaging, surgical robotics, image guided therapies and surgical process analysis. I first studied electronics and computer systems engineering at King's College before completing a PhD in computer science at Imperial College specializing in medical image computing.