Towards automatic skill evaluation: Detection and segmentation of robot-assisted surgical motions

Figures & data

Figure 1. The eight rudimentary surgical gestures, or surgemes, common in a four-throw suturing task, as defined by a senior cardiac surgeon. 1) Reach for needle. 2) Position needle. 3) Insert and push needle through tissue. 4) Move to middle with needle (left hand). 5) Move to middle with needle (right hand). 6) Pull suture with left hand. 7) Pull suture with right hand. 8) Orient needle with both hands. [Color version available online.]

Table I.  The 72-data-point subset used in the study. The full 192-data-point da Vinci API also contains Cartesian positions, rotation matrices and other data.

	Cartesian velocities (6)	xvel, yvel, zvel, xrot, yrot, zrot
Master console manipulators (2)	joint positions (8)	joint1, joint2, …, joint7, gripper
	joint velocities (8)	joint1, joint2, …, joint7, gripper
Patient-side manipulators (2)	joint positions (7)	joint1, joint2, …, joint7, gripper
	joint velocities (7)	joint1, joint2, …, joint7, gripper

Figure 2. Functional block diagram of the system used to recognize elementary surgical motions in this study.

Figure 3. Creating a super feature vector that encapsulated temporal information from neighboring data samples. This example assumes a subsampling granularity of s = 2. [Color version available online.]

Figure 4. Cartesian position plots of the da Vinci left master manipulator, identified by motion class, during performance of a 4-throw suturing class. The left plot (a) depicts the data for the expert surgeon while the right (b) represents the intermediate surgeon. [Color version available online.]

Figure 5. Recognition rates of the four motion class definitions for the expert surgeon across all 14 temporality sizes with window removal sizes of w = (3, 5, 7). Motion class 11234455 had the highest average recognition rate across all 3 dimensions. [Color version available online.]

Figure 6. Recognition rates of the six motion class definitions for the intermediate surgeon across all 14 temporality sizes with window removal sizes of w = (3, 5, 7). Motion class 12344567 had the highest average recognition rate across dimensions 4 and 5, while 12233456 had the highest rate for dimension 3. [Color version available online.]

Table II.  The four motion classes used for the expert data. Each motion class defines the mapping from the surgeme in the vocabulary to the corresponding motion in the class.

Motion class definition	Surgeme
	1	2	3	4	5	6	7	8
1	1	2	3	4	5	5	6	6
2	1	2	3	4	5	5	5	5
3	1	1	2	3	4	4	5	5
4	1	2	2	3	4	4	4	4

Table III.  The six motion classes used for the intermediate data. Each motion class defines the mapping from the surgeme in the vocabulary to the corresponding motion in the class.

Motion class definition	Surgeme
	1	2	3	4	5	6	7	8
1	1	2	3	3	4	4	5	5
2	1	2	3	3	4	4	4	4
3	1	1	2	2	3	3	4	4
4	1	2	3	3	4	5	6	7
5	1	2	3	4	4	5	6	7
6	1	2	2	3	3	4	5	6

Figure 7. Having a temporal length m = 8 and a subsampling granularity of s = 2 resulted in the highest average recognition rate. Each pair was tested across all 4 motion classes, all 3 dimensions, and on transition removal sizes of 3, 5 and 7. Note that not using any temporal information resulted in the lowest average recognition rates.

Figure 8. Having a temporal length m = 16 and a subsampling granularity of s = 2 resulted in the highest average recognition rate. Each pair was tested across all 6 motion classes, all 3 dimensions, and on transition removal sizes of 3, 5 and 7. Note that not using any temporal information resulted in the lowest average recognition rates.

Table IV.  The various combinations of temporal region m and subsampling rate s. The ordering is based on increasing temporal window size (sm + 1).

	Neighbors (m)	Subsampling rate (s)
1	1	1
2	2	1
3	3	1
4	5	1
5	3	2
6	4	2
7	10	1
8	5	2
9	32	2
10	10	2
11	12	2
12	16	2
13	20	2
14	24	2

Figure 9. The result of LDA reduction with m = 6 and d = 3. The motions of the expert surgeon (left) separate more distinctly than those of the intermediate surgeon (right). [Color version available online.]

Figure 10. Comparison of automatic segmentation of robot-assisted surgical motion with manual segmentations. Note that most errors occur at the transitions. [Color version available online.]

Figure 11. Results of varying window removal size across all temporality sizes and motion classes, and with LDA dimension 4 for the expert data. Removal of a window size of 5 or 7 returned the highest recognition rates. [Color version available online.]

Figure 12. Results of varying window removal size across all temporality sizes and motion classes, and with LDA dimension 5 for the intermediate data. Removal of a window size of 5 or 7 returned the highest recognition rates. [Color version available online.]

Table V.  Results of training on both the expert and intermediate data sets, which barely decreased the recognition rates of the classifier. A motion class definition of 12344455, with temporal size of m = 8 and s = 2 and LDA dimension of 5 was used. The comparison parameters for the standalone are expert = {11234455, m = 8, s = 2, d = 5} and intermediate = {12344567, m = 16, s = 2, d = 5}.

Preliminary results on combined datasets
Window Size	0	1	3	5	7
Recognition rate (%)	88.46	89.27	90.74	91.72	91.87
Comparison to standalone results
Expert	88.53	89.33	90.96	91.94	92.21
Intermediate	91.71	92.58	93.99	94.91	95.26