Real-time tracking of surgical instruments based on spatio-temporal context and deep learning

Figures & data

Figure 1. The training and validation error of the CNN, (a) our network without the STN; (b) our network with the STN.

Figure 2. Specific network architecture of (a) the STN and (b) the classification network.

Figure 3. (a) Learning spatial context at the $t_{th}$ frame; (b) detecting an object’s location at the ${(t+1)}_{th}$ frame and tracking.

Figure 4. Test video images, from top to bottom, with rows showing the performance of our method, the MIL method, ATF method, DDVT method, and CF method on eight datasets, respectively. Every column represents one kind of dataset; the dataset names appear below each column.

Figure 5. Each graph represents one kind of dataset and each colour line represents each method’s error; the dataset names appear below each graph. Red represents the error of our method, green that of the CF method, yellow that of the MIL method, black that of the ATF method, and blue that of the DDVT method.

Table 1. Mean tracking errors (pixel) of every method on every dataset.

Dataset Algorithms	In vivo Abdominal condition	Laparoscopic simulation condition	Light- Changing condition	Retinal condition	Smoky condition	Cardiac condition	Our own simulation condition	Standard dataset
DDVT [6]	41.98	45.40	28.29	21.18	28.47	N/A	40.46	37.05
MIL [22]	N/A	N/A	45.71	9.02	32.08	N/A	N/A	N/A
CF [20]	14.40	13.47	25.64	10.45	18.98	N/A	35.21	39.43
ATF [10]	N/A	N/A	45.37	36.85	55.33	N/A	N/A	N/A
Ours	15.35	12.44	17.58	13.71	16.86	N/A	14.42	11.31

Table 2. Standard deviations (pixel) of every method’s tracking errors on every dataset.

Dataset Algorithms	In vivo Abdominal condition	Laparoscopic simulation condition	Light- Changing condition	Retinal condition	Smoky condition	Cardiac condition	Our own simulation condition	Standard dataset
DDVT [6]	21.77	70.50	27.61	17.22	12.26	N/A	52.04	64.77
MIL [22]	N/A	N/A	22.27	3.98	13.91	N/A	N/A	N/A
CF [20]	8.17	7.61	17.42	5.32	7.95	N/A	4.18	8.55
ATF [10]	N/A	N/A	22.38	12.41	17.06	N/A	N/A	N/A
Ours	13.67	19.14	20.52	6.81	9.97	N/A	2.63	9.42

Table 3. The trackers’ accuracy on every dataset; the accuracy have been calculated based on the distance between the ground truth and the tracking position. If the distance is smaller than the half length of side of the tracking window centred on the ground truth in one frame, we believe the tracker successfully track the object in this frame.

Dataset Algorithms	In vivo Abdominal condition	Laparoscopic simulation condition	Light- Changing condition	Retinal condition	Smoky condition	Cardiac condition	Our own simulation condition	Standard dataset
DDVT [6]	87%	89%	90%	90%	98%	5%	94%	99%
MIL [22]	74%	17%	73%	100%	40%	5%	40%	96%
CF [20]	96%	99%	96%	100%	100%	9%	99%	100%
ATF [10]	10%	35%	88%	95%	23%	5%	22%	79%
Ours	93%	96%	99%	100%	100%	82%	100%	100%

Table 4. Every method’s frames per second (FPS) on every dataset.

Dataset Algorithms	In vivo Abdominal condition	Laparoscopic simulation condition	Light- Changing condition	Retinal condition	Smoky condition	Cardiac condition	Our own simulation condition	Standard dataset
DDVT [6]	0.61	0.57	0.60	0.60	0.60	N/A	0.63	0.62
MIL [22]	0.19	0.26	0.27	0.22	0.26	N/A	0.26	0.21
CF [20]	0.64	0.72	0.64	0.61	0.67	N/A	1.59	1.34
ATF [10]	N/A	2.64	4.54	1.80	2.76	N/A	2.15	2.40
Ours	33.19	37.89	34.25	32.41	40.58	5.20	35.61	25.10