Tracking-by-detection of surgical instruments in minimally invasive surgery via the convolutional neural network deep learning-based method

Figures & data

Figure 1. The operation part of MIS instrument: end-effector with the tip point and shaft with the insertion point.

Figure 2. Line scanner application for detection of shaft edge lines and shaft image direction estimation.

Figure 3. The CNN architecture for the end-effector detection consisting of 5 convolutional (conv1 ∼ 5) and 2 fully connected (fc6 ∼ 7) layers.

Figure 4. The 2D trajectories of end-effector tip obtained by the proposed method (solid lines) and that in [7] (dash-dotted lines).

Figure 5. The end-effector tip 3D trajectories in X-Y-Z coordinates.

Figure 6. The end-effector tip 3D errors in the X-Y-Z coordinates.

Figure 7. Selected frames of the instrument tracking and detection: the red circles are the tracked end-effector tip position, and the green dashed line is the shaft symmetry axis.

Table 1. The mean errors and standard deviations for 2D/3D numerical results obtained by the proposed method.

	3D errors (mm)			2D errors (pixel)
	X	Y	Z	u	v
Mean	2.5697	1.9701	0.3545	3.1417	2.7073
Std. Dev.	0.8172	0.8076	0.1984	3.7129	3.0085

Figure 8. Example frames of in-vivo sequences with the end-effector positions shown by squares.

Table 2. The 2D mean errors and standard deviations of in-vivo test.

	2D errors (pixel)
	u	v
Mean	6.1695	5.2378
Std. Dev.	5.9261	5.4736

Zhao Z. Real-time 3D visual tracking of laparoscopic instruments for robotized endoscope holder. Bio-med Mater Eng. 2014;24:2665–2672.

 PubMed Web of Science ®Google Scholar