Providing metrics and performance feedback in a surgical simulator

Figures & data

Figure 1. A. The surgeon is not maintaining proper visibility of the drilling region because it is obstructed by a shelf of bone, and will be unable to see that he/she is nearing the facial nerve. B. Correct exposure of the facial nerve has been achieved, as all but a thin layer of bone has been removed above it.

Figure 2. Force magnitudes applied by experts and novices as functions of distance from the facial nerve. Data points were sorted into bins of width 0.2 cm based on distance from the structure. The means of each bin are plotted, along with error bars showing standard deviations within the bins. The data was collected in the user study described in the Validation section. Experts applied smaller forces than the novices and decreased their forces as they approached the facial nerve. [Color version available online.]

Figure 3. An overview of the metrics console. A video is replaying in the left panel in sync with a model reconstruction of the virtual environment in the right panel. The currently selected metric is amount of accumulated bone dust. An interval of relatively high accumulation is being viewed, as shown by the frame position scroll bar located below a peak in the graph of the time variation of the currently selected metric (red dots). [Color version available online.]

Table I.  Time taken by different tasks when opening the performance evaluation console. The times given are averages based on 32 data files collected in the user study described in the Validation section.

	Time		Time
Actual procedure	844	Exposure	8.34
Total startup time	38.5	Direct and overexposure	7.89
Load anatomy	15.2	Inferred	0.446
Read data file	6.01	Visibility	0.845
Compute metrics	17.2	All other metrics	0.0604
Mesh collisions	4.46	Building metric graphs	0.0211
Bone dust	2.79	Reset simulation	0.0772

Figure 4. An overview of the console, with the visibility metric selected. It is clear that the user used proper saucerizing technique while making the hole on the right but is now undercutting while drilling the hole on the left. Red dots in the visually annotated reconstructed model view show where the mistake occurred, and the metric-versus-time graph below the display panels shows when it occurred. [Color version available online.]

Figure 5. An example of interactive feedback in the simulator, with the bone removal metric selected. Bone removed by the user at the far right that was removed by less than 20% of the experts (as parameterized in the “Prob” text box in the bottom right corner) is shown in red, while bone in the center that was removed by a higher percentage of the experts is shown in green. The bar at left shows the relative proportion of correctly (green) and incorrectly (red) removed bone. [Color version available online.]

Table II.  Average proportion correctly classified using HMMs with different numbers of states and different sets of features.

	3	4	5	6	7	8
Force	0.681	0.700	0.713	0.744	0.763	0.775
Force, position, distance	0.750	0.769	0.750	0.806	0.756	0.731
Position	0.750	0.538	0.631	0.619	0.631	0.656
Force, position, distance, suction position	0.788	0.744	0.769	0.825	0.850	0.800
Force, position, distance, suction-drill distance	0.738	0.788	0.819	0.769	0.819	0.813

Table III.  Correlations of metrics with the average of the global scores assigned by the two instructors.

Metric	r	p
Drilling technique
Pct bone visible at removal (1)	0.728	<0.001
Pct removed with burr too large (2)	−0.405	0.022
Jump frequency (3)	−0.226	0.213
Mean drill angle (4)	−0.405	0.022
Suctioning technique
Pct excessive inter-tool distance (5)	−0.469	0.007
Pct excessive dust (6)	−0.365	0.040
Bone removal
Mean removal probability (7)	0.337	0.059
Pct improbable (non)removals (8)	−0.794	<0.001
Facial nerve exposure
Pct directly exposed (9)	0.469	0.007
Pct direct or indirect exposed (10)	0.519	0.002
Pct overexposed (11)	−0.536	0.002
Drill forces
Pct excessive force (12)	−0.355	0.046
Near dura (13)	−0.471	0.007
Near sigmoid (14)	−0.420	0.017
Near facial nerve (15)	−0.563	<0.001
Near inner ear (16)	−0.468	0.007
Drill Velocities
Pct excessive velocity (17)	−0.131	0.474
Near dura (18)	−0.152	0.407
Near sigmoid (19)	−0.143	0.434
Near facial nerve (20)	−0.387	0.029
Near inner ear (21)	−0.339	0.058

Table IV.  Correlations of metrics with global scores and with metric-specific sub-scores.

	Specific score		Global score
Metric	r	p	r	p
Pct bone visible at removal (1)	0.777	<0.001	0.728	<0.001
Jump frequency (3)	−0.355	0.046	−0.226	0.213
Pct excessive distance between tools (5)	−0.737	<0.001	−0.469	0.007
Pct excessive dust (6)	−0.681	<0.001	−0.365	0.040
Mean removal probability (7)	0.323	0.072	0.337	0.059
Pct improbable choices of removal regions (8)	−0.736	<0.001	−0.794	<0.001
Pct of facial nerve directly exposed (9)	0.400	0.023	0.469	0.007
Pct of facial nerve directly or indirectly exposed (10)	0.411	0.019	0.519	0.002
Pct of facial nerve overexposed (11)	−0.500	0.004	−0.536	0.002

Table V.  Percentage of experts and novices correctly classified with leave-one-out cross-validation using a logistic regression classifier with individual metrics as features.

	Proportion correct
Metric	Expert	Novice	Overall
Pct bone visible at removal	1.000	1.000	1.000
Improbable nonremovals	0.875	0.875	0.875
Pct excessive inter-tool distance	0.875	0.875	0.875
Pct exc forces near facial nerve	0.875	0.875	0.875
Pct removed with burr too large	0.875	0.750	0.813
Pct overexposed facial nerve	1.000	0.625	0.813
Mean removal probability	0.750	0.750	0.750
Improbable removals	0.750	0.750	0.750
Pct directly exposed facial nerve	0.875	0.625	0.750
Average drill angle deviation from 90^°	0.875	0.625	0.750
Pct excessive velocities near facial nerve	0.875	0.500	0.688
Pct excessive dust	0.750	0.625	0.688
Pct excessive velocities	0.750	0.500	0.625
Number of collisions	0.875	0.375	0.625
Jump frequency	0.750	0.500	0.625
Pct removed with burr too small	0.875	0.375	0.625
Pct indirectly exposed facial nerve	0.750	0.500	0.625
Pct overexposed chorda	0.875	0.375	0.625
Pct overexposed sigmoid	1.000	0.125	0.563
Pct excessive forces	0.375	0.625	0.500

Figure 6. Reviewing a procedure in the performance evaluation console study. Members of the Feedback Group received interactive feedback about their maintenance of visibility with red (poor) and green (good) dots as bone was drilled away and a meter showing the percentage of red and green voxels over the last 50 removals, as shown in the video replaying in the left panel. After completing a trial, they were able to review their performance with respect to visibility, exposure/overexposure, and collisions with the tube. Shown at right is a visualization of achieved exposure, with properly exposed regions in white, overexposed regions in red, and unexposed regions in blue. [Color version available online.]