Challenges in surgical video annotation

Figures & data

Table 1. Overview of some of the challenges, key points, and recommendations to overcome those challenges in surgical video annotation.

Challenge	Key challenge points	Overcoming the challenge
Varying clinical expertise of annotators and cost of clinical experts	Different clinical phenomena require different levels of clinical expertise to identify	Determine clinical complexity of the phenomena to be annotated and determine appropriate relevant clinical expertise ‘Two-pass’ annotation with clinical experts verifying or augmenting annotations of trainees or laypersons
Poor inter-annotator reproducibility (i.e. high variability amongst annotators)	Multiple annotators may be imprecisely annotating clinical phenomena, resulting in significantly variability in annotations Multiple annotators may not conceptualize clinical phenomena in a similar manner	Precise definitions of clinical phenomena via annotation guides Careful selection of appropriate metrics of reliability to assess quality of annotations Qualitative review of variable annotations may identify which situations lead to variability due factors such as clinical difficulty
Incomplete representation of spatial clinical phenomena	Annotations of spatial data may be biased toward clear situations that are not clinically reproducible	Establishment of clear a priori clinical phenomenon of interest for spatial annotation
Selection of identification vs. segmentation methods	Spatial annotations with bounding boxes may incompletely capture the clinical phenomenon of interest Spatial annotations with semantic annotation may be too time-consuming for the clinical phenomenon of interest	Clearly establish clinical phenomenon of interest on which to base annotation strategy
Selection of causal vs. acausal methods of workflow analysis	Performance in workflow analysis can be higher with acausal approaches but may limit clinical utility	Define the intended clinical use case for the proposed work – decision-making and other uses that require online analysis require causal approaches while posthoc uses may benefit from acausal approaches
Annotation of clinically meaningful events	Defining ‘clinically meaningful’ events can be difficult as different clinicians have different conceptions of what is meaningful	Engage in a priori discussion with clinical experts to define clinically meaningful events in the context of the research project
Assessment of surgical performance has few objective measures	Existing performance assessments such as OSATS are subjective and require rater training for good inter-rater reliability	Performance should ultimately be assessed by patient outcomes
Expanding types of annotations can make existing annotations incomplete or obsolete	An existing set of annotations is unlikely to be exhaustive and may require updating	Version control enables layering of annotations to improve, update, or augment existing labels

Table 2. Annotator classes and relevant clinical experience.

Clinical annotator class	Experience
Clinical Expert	Completed residency/fellowship
Clinical Trainee	In residency training
Layperson	No clinical background/training
Crowd	Crowd platform based (e.g. Mechanical Turk) groups of annotators, typically without clinical background/training

Figure 1. Comparison of two different views of the hepatocystic triangle illustrating different levels of dissection that can be performed in attempting to obtain a critical view of safety.

Figure 2. Use of bounding boxes (left) may result in overlapping identification of structures versus semantic segmentation (right). Consideration should be given to which approach more faithfully reflects the clinical phenomenon of interest. Green denotes gallbladder, blue denotes cystic duct, orange denotes cystic artery.

Table 3. An excerpt of part of the Objective Structured Assessment of Technical Skills from Martin et al. [28].

Time and motion
1	2	3	4	5
Many unnecessary moves		Efficient time/motion but some unnecessary moves		Clear economy of movement and maximum efficiency
Flow of operation
1	2	3	4	5
Frequently stopped operating and seemed unsure of next move		Demonstrated some forward planning with reasonable progression of procedure		Obviously planned course of operation with effortless flow from one move to the next

Martin J, Regehr G, Reznick R, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997;84(2):273–278.

 PubMed Web of Science ®Google Scholar