Visual gaze patterns in trainee endoscopists – a novel assessment tool

Abstract

Background

Colonoscopy proficiency is significantly influenced by skills achieved during training. Although assessment scores exist, they do not evaluate the impact of visual search strategies and their use is time and labour intensive. Eye-tracking has shown significant differences in visual gaze patterns (VGPs) between expert endoscopists with varying polyp detection rates, so may provide a means of automated assessment and guidance for trainees. This study aimed to assess the feasibility of eye-tracking as a novel assessment method for trainee endoscopists.

Methods

Eye-tracking glasses were used to record 26 colonoscopies from 12 endoscopy trainees who were assessed with directly observed procedural scores (DOPS), devised by the Joint Advisory Group (JAG) on GI endoscopy, and a visual analogue score of overall competence. A ‘total weighted procedure score’ (TWPS) was calculated from 1 to 20. Primary outcomes of fixation duration (FixD) and fixation frequency (FixF) were analysed according to areas of interest (AOIs) with the bowel surface and lumen represented by three concentric rings. Correlation was assessed using Pearson’s coefficient. Significance was set at p<.050.

Results

Trainees displayed a significant positive correlation between TWPS and FixD (R = 0.943, p<.0001) and FixF (R = 0.936, p<.0001) in the anatomical bowel mucosa peripheries. Conversely, they had significant negative correlations between TWPS and the anatomical bowel lumen (FixD: R= −0.546, p=.004; FixF: R= −0.568, p=.002).

Conclusions

Higher objective performance scores were associated with VGPs focussing on bowel mucosa. This is consistent with prior analysis showing peripheral VGPs correspond with higher polyp detection rates. Analysis of VGPs, therefore, has potential for training and assessment in colonoscopy.

Keywords:

• Visual gaze pattern
• eye-tracking
• colonoscopy training

Introduction

Effective colonoscopy is dependent on thorough examination of the mucosa to allow detection and excision of polyps. However, this is limited by variations in skill level amongst endoscopists resulting in missed lesions [1–3]. The adenoma detection rate (ADR) is an established indicator of performance and has shown to impact colorectal cancer (CRC) risk and mortality [4–8], with studies showing a 3% decrease in CRC risk for every 1% increase in ADR [9]. Studies investigating inter-operator disparity in ADRs report a correlation with individual skill levels [10,11] and significant influence from experience during training. Van Doorn et al. demonstrated that endoscopists with low ADRs during training are more likely to have low ADRs in independent practice [12]. However, UK-based trainee endoscopists are not formally assessed with ADRs as it is the responsibility of the trainer to ensure the examination is completed properly, so this becomes part of the trainers’ record.

The ADR reflects both the manual skill of the endoscopist and their visual search strategy to identify lesions. Currently, trainees learn to perform colonoscopies through guided supervision until they are deemed independent. To assess their skills several validated objective assessments have been created such as the Joint Advisory Group (JAG) on GI Endoscopy Training System Directly Observed Practical Skills (JETS DOPS) [13–17]. Although these tools allow analysis of manual competency, they do not assess visual search strategies and are time and labour intensive. This has meant that the JETS DOPS score is limited to use in education and research. Therefore, a novel automated objective assessment tool is required to eliminate the drawbacks of current assessment systems and ensure adequate performance from an early stage during training.

Eye-tracking is the measurement of eye movements which can be statistically analysed to determine the pattern and duration of eye gaze fixations. It maps the user’s visual focus onto their field of view and can accurately record visual gaze patterns (VGPs), orientation and perception of anatomical landmarks during complex psychomotor tasks [18]. Our group has used eye-tracking to investigate visual areas of interest (AOIs) between expert and novice surgeons in live open and laparoscopic surgery [19–25]. Within surgery, knowledge and identification of key anatomical landmarks is essential for orientation and safe practice. Therefore, it is assumed that these AOIs become more defined and occupy most of the gaze time with increasing expertise [26].

The use of eye-tracking in endoscopy has revealed significant differences in VGPs between different endoscopists [27] as well as differences in search strategies between experts and novices [28] and between expert endoscopists with varying polyp detection rates [29]. However, little is known about how these VGPs develop during training. As trainees do not record individual polyp or ADRs, it is important to be able to assess and coach them from an early stage to ensure optimum performance when practicing independently. This study, therefore, aims to investigate VGPs in trainee endoscopists during the withdrawal phase of colonoscopies and correlate these with DOPS Scores, with a view to assessing the feasibility of eye-tracking as an objective assessment and training tool in colonoscopies.

Materials and methods

This prospective observational study was conducted within three Endoscopy Units across Imperial College Healthcare NHS Trust. The study was performed with ethical approval from the West London & GTAC Research Ethics Committee on 5 August 2014, reference 13/LO/0119.

Participants

Trainee endoscopists who were able to conduct a colonoscopy from initiation until completion were recruited. Demographic data including number of previous colonoscopies and stage of training was collected. All patients undergoing a whole bowel colonoscopy between March and May 2019 were considered for inclusion. Patients with established inflammatory bowel disease and/or inadequate bowel preparation were excluded from the study. Poor bowel preparation could contribute to changes in search strategies and introduce bias, thus only colonoscopies with a score of 2 or 3 on the Boston Bowel Preparation Scale were included.

Data collection

Endoscopy trainees were equipped with lightweight eye-tracking glasses (model: Tobii Pro Glasses 2, Tobii Pro Sweden) prior to starting a procedure. The glasses incorporated a forward-facing wide-angle scene camera to record the endoscopy screen and four internal infrared cameras to record pupil centre corneal reflections via dark pupil tracking. One-point calibration, automatic slippage compensation and accelerometer and gyroscope sensors allowed the pupil position to be tracked in relation to the field of view even whilst moving. The glasses had a binocular sampling rate of 100 Hz (50 Hz at extreme viewing angles) with a visual field range of 82° horizontally & 52° vertically. Data from the glasses were live streamed to the Tobii Glasses Controller Software (Tobii AB, Stockholm, Sweden), which superimposed the endoscopist’s gaze focus onto their field of view.

Recordings were taken during the withdrawal phase of the colonoscopy, starting from the caecum. Segments that were associated with flushing, polyp biopsy and removal, and patient positioning were excluded until the first 2 min of polyp search data was isolated for each participant. Throughout data collection, the researcher was present to assess for distractions irrelevant to the colonoscopy. Coaching of the participant was limited unless it was deemed to have an impact on patient safety or outcomes.

Objective assessment of endoscopists’ skill

Eye-tracking metrics were compared with validated DOPS assessments from the JETS e-Portfolio. Other well-used assessment tools include Global Assessment of Gastrointestinal Endoscopic Skills in Colonoscopy (GAGES-C) and the Mayo Colonoscopy Skills Assessment Tool (MCSAT). Although both GAGES-C and JETS e-portfolio are previously validated objective assessment tools for endoscopic skill [13,16], the JETS e-portfolio is also a national requirement for all endoscopy trainees and provides information on key performance indicators such caecal intubation rate, polyp detection rate and whether the trainee is ready for independent practice [14,15]. JETS has been validated by JAG, and additional analysis of learning curve cumulative summation using the JETS database has shown 41% of trainees to be competent after 200 procedures [16]. MCSAT is not currently used in accredited assessment [17], unlike GAGES-C and JETS.

Impartial assessment of trainees’ colonoscopy skill was conducted by expert independent colonoscopists, each of whom had performed over 1000 colonoscopies, during the live colonoscopy procedure. The experts were blinded to trainees’ VGPs during the procedure. Trainees were graded according to JETS DOPS assessments where a score of 1 was given for ‘maximal supervision’ and 4 for ‘competent for independent practice’. The maximum number of points available from the ‘procedure score’ section of the DOPS, which was most relevant for colonoscopy technique, was 32. A visual analogue scale was also used to give an ‘overall assessment’ from 1 (maximal supervision) to 10 (independent) as shown in Figure 1. This was used to reflect the differences in difficulty level of various colonoscopies in the overall score. These were then used to derive a ‘Total weighted procedure score’ (TWPS) with equal weighting on the ‘overall assessment’ and ‘DOPS procedure score’. The equation used for the TWPS was $\frac{\text{DOPS procedure score}}{3.2}+\text{Overall assessment score}$ to give a score from 1 to 20.

Figure 1. (A) Procedure score section of the validated DOPS Score. A score of 1 was given for 'maximal supervision' and 4 for 'competent for independent practice' with a total of 32 points. (B) Visual analogue score from 1 to 10.

Eye-tracking metrics

The primary outcome measures were standardised fixation duration (FixD) and fixation frequency (FixF) according to each AOI. FixD is the total amount of time each participant fixated on each AOI in seconds. FixF is the number of fixations within each AOI in fixations per second. The secondary outcome measure was maximum pupil diameter throughout the withdrawal.

Eye-tracking metrics were analysed according to AOIs relating to the bowel anatomy. Three concentric circles represented the bowel anatomy (Figure 2): inner circle – lumen, middle circle – edge of lumen and outer circle – rest of the colonic mucosa. A screen-based analysis was also conducted with the results available upon request.

Figure 2. (A) Areas of interest – the bowel was anatomically divided into three concentric rings representing the bowel mucosa (outer ring), edge of the lumen (middle ring) and lumen (inner circle). This was further divided into segments labelled from 1 to 17 and analysed in various regional combinations. (B) Image demonstrating the areas of interest mapped onto the colonoscopy screen.

Analysis

The Tobii Pro lab software (Tobii AB, Stockholm, Sweden) was used to manually map fixations from the recordings onto a still image according to AOIs for the anatomical analysis by an individual researcher to ensure consistency. These mapped fixations were then used to generate eye-tracking metrics on FixD (seconds) and FixF (count/s) with respect to percentage of total time spent of each individual AOI throughout the 2 min of withdrawal. The secondary outcome measure of maximum pupil diameter (mm) was also generated. Shapiro–Wilk tests showed the data was non-parametric, thus correlation between primary and secondary outcomes with DOPS scores was calculated using Pearson’s coefficient. All statistical analyses were conducted using SPSS^® version 25.0.0.0 (IBM, Armonk, NY). Significance was set at p<.050.

Results

A total of 29 individual colonoscopies from 12 trainee endoscopists were recorded. Of these, three (10.3%) recordings were removed from analysis due to poor glasses calibration during data recording. A summary of endoscopist demographic data is displayed in Table 1.

Table 1. Demographic data from trainee endoscopists included in the study.

Years of specialist training	Number of endoscopists	Number of previous colonoscopies (mean)	Standard deviation	Total number of recordings
1	1	100	n/a	3
2	4	141	118.77	11
3	2	53	24.75	2
4	2	245	21.21	4
5	3	510	200.75	6

Objective skills assessment

There was a significant positive correlation between trainees with higher TWPS and greater FixD in the outer circle (Figure 3), which represented the bowel mucosal surface (R = 0.943, p<.0001). This corresponded with significant negative correlations between TWPS and FixD on the middle circle (R= −0.914, p<.0001) and the lumen (R= −0.546, p=.004). There was also a significant positive correlation between TWPS and the AOI ‘5,13′ (R= 0.529, p=.005). No significant correlations were found with other AOIs (p>.050).

Figure 3. Graph showing correlation between total weighted procedure score and fixation duration (%) in the outer circle (R = 0.943, p<.0001).

FixF followed a similar pattern of results as FixD with a significant positive correlation between trainees with higher TWPS and FixF in the outer circle (R = 0.936, p<.0001) (Figure 4) and significant negative correlations between TWPS and FixF on the middle circle (R= −0.919, p<.0001) and lumen (R= −0.568, p  =.002). There were also significant positive correlations between TWPS and the AOI ‘5,13′ (R = 0.484, p=.012) and the AOI ‘3,4,5,11,12,13′ (R= 0.437, p=.026). No significant correlations were found with other AOIs (p>.050).

Figure 4. Graph and heat maps showing a positive correlation between total fixation frequency on the outer ring, which represented the walls of the colonic mucousa, and the total weighted procedure score (R = 0.936, p<.0001). The heat maps show the cumulative distribution of fixation frequency for (i) the lowest scoring 8 colonoscopies, (ii) the middle scoring 10 colonoscopies, and (iii) the highest scoring 8 colonoscopies (from bottom left to top right of the graph).

Pupil diameters

There was a significant negative correlation between maximum pupil diameter in the right eye and TWPS (R= −0.518, p=.007) (Figure 5). No significant correlation was found between TWPS and left pupil maximum diameter (p>.050).

Figure 5. Graph showing correlation between total weighted procedure score and right pupil maximum diameter (mm) (R= −0.518, p=.007).

Discussion

This is the first study which establishes an association between VGPs and skill levels of individual endoscopists based on a validated objective assessment scale. It, therefore, identifies the potential use of eye-tracking as an objective assessment tool during colonoscopy training. A high FixD is understood to represent higher cognitive attention through ignoring distracting stimuli to focus on the AOIs relevant to the task [30,31]. Similarly, FixF is used to determine which AOIs are most important to endoscopists through frequent visitations. Colonoscopy trainees with higher TWPS had greater FixD and FixF along the peripheries of the bowel mucosa indicating a greater importance placed on these AOIs. Conversely, for lower-skilled trainees FixD and FixF were greatest at the lumen and lumen edge.

The increasingly peripheral search patterns associated with higher skill levels may represent a more thorough and systematic search strategy. This would cover a larger surface area of bowel mucosa when looking for abnormalities. Traditionally endoscopists prefer to manipulate the endoscope so that a segment of bowel is completely in view [32,33]. This endoscopic manipulation is also shown by Dik et al. [27] where regions of fixation between the anatomical colonic surface and endoscopy screen strongly correlated. This data in this study suggest that trainees who are more skilled and comfortable with endoscopic manoeuvring, have a more economical and effective range of eye movements to focus on searching for lesions instead of fixating on the lumen to correct endoscope position.

Similar VGPs have been seen in previous studies where expert endoscopists demonstrated more peripheral search patterns on the screen when looking for lesions whilst viewing a video of a colonoscopy compared to novices [28]. Notably, a previous study by our group found expert endoscopists with higher polyp detection rates had higher fixation frequencies and durations on the outer ring [29], thereby affirming this VGP as a more optimum search strategy. An interesting comparison is with VGPs whilst driving and searching for hazards. Experienced drivers were found to gaze into the distance and scan in all directions for oncoming traffic, whereas novices focussed in front of the car [34]. The hypothesis being that experienced drivers appreciated where hazards were most likely to come from. Therefore, with the use of eye-tracking, it would be possible to assess trainees from an early stage, so they can learn to focus on systematically searching bowel areas important in detection of lesions. This can not only accelerate their learning curves, but also enable early development of optimum VGPs that can be carried on to independent practice.

Lower scoring trainees were also found to have a weak correlation with lower FixD and FixF in the bottom right corner of the bowel wall. This may also reflect endoscopic manoeuvring with trainees attempting to maintain the field of view which disappears to the left during withdrawal whilst going past the hepatic flexure, as demonstrated in previous work from our group and others [28,35]. Lower scoring trainees also had a weak correlation with larger maximum pupil diameter in the right eye across the entire withdrawal phase. Hess and Polt first investigated pupil diameter and found it was positively correlated with high cognitive workload [36]. This has since been validated through various studies in different fields [37–39]. This suggests that higher-scoring trainees may experience lower cognitive workloads during the procedure. However, formal cognitive scoring would be required to confirm this hypothesis. When combined with the more central search strategies, this supports the hypothesis that lower-scoring trainees require more effort to manoeuvre the endoscope to maintain adequate visualisation than higher scoring trainees, who are then able to concentrate more on search strategies.

The limitations of this study must be considered. Although the multi-centre methodology was beneficial in assessing trainees of various skill levels, it risked variability in supervisor scoring. Since this was recognised, the JETS ePortfolio DOPS was used as it is a national requirement that has been standardised over a decade of use [15]. As UK-based trainee endoscopists are not given ADRs, it was not possible to directly correlate their VGPs with clinical outcomes. Additionally, it was not possible to assess adequate mucosa visualisation, including those not easily visualised due to be obscured by haustral folds. Moreover, to compensate for variability in procedure complexity, supervisors also provided a visual analogue score. Unfortunately, the study contained no sample size calculation as it the first to analyse gaze data within a cohort of trainee endoscopists. This study had a small sample size and larger studies shall be necessary to confirm the findings of the present analysis and validate the technology further. Whilst the analysed segments were limited to 2 min to ensure standardisation in anatomy and timing, it is unlikely that this would be wholly representative of VGPs that are sustained during manoeuvring through the rest of the colon, particular the flexures. Assessment of VGPs throughout colonoscopy will be required to further validate eye-tracking as an assessment tool. During the study, trainees may have been subject to the Hawthorne effect, whereby a subject’s behaviour changes when they are observed. However, Nielsen et al. showed this does not appear to change search strategies with no change in ADR despite increased withdrawal times [40]. Also, all participants would have been subject to the same effect.

Currently, colonoscopy training focuses on techniques such as patient positioning and rotating of the endoscope to increase visualisation of the bowel mucosa. There is no standard approach to address the optimum method of searching the anatomical surface or screen for abnormalities [41,42]. Trainees are therefore reduced to ‘discovery’ learning where they experiment and learn from mistakes or through advice from supervisors and peers [43]. This leads to heterogenous visual search patterns and contributes to the variability in ADRs. However, along with providing an objective measure of search strategies, eye-tracking technology could be used to enhance and standardise colonoscopy training.

The characterisation of experts’ VGPs within laparoscopic surgery has led to its use in gaze training through identification of AOIs [21]. This has been shown to improve the efficacy of laparoscopic procedures by verbally guiding trainees to focus on AOIs or through mapping AOIs from the trainer onto the screen for visual guidance [44]. For example, Vine et al. taught expert gaze strategies to novices performing simulated laparoscopic tasks and found faster completion times and fewer errors than those novices without gaze training [45]. While this study implies that a simple recommendation to trainees may be to consciously fixate on the periphery of the view rather than the lumen, further detailed studies of expert endoscopists with high ADRs may help define more detailed optimum VGPs during withdrawal, and therefore help with endoscopy training.

In conclusion, colonoscopy trainees of varying skill levels based on a validated objective assessment scale also demonstrate differences in their VGPs. Higher scoring trainees focus more on the periphery of the screen which may reflect a more systemic search strategy for abnormalities. They also demonstrate decreased cognitive workload. However, whilst this study establishes a correlation between technical skills and VGPs, it is not possible to prove an independent association.

Compared with other validated skills assessment methods such as global rating scales and motion analysis, eye-tracking is still very much in the early stages of research. However, it has the advantages of being able to measure real-time performance through visualisation of the trainees’ eye movements on a screen utilising machine learning techniques and can also be used in re-validation of consultants. Moreover, given the advancements in eye-tracking technology, glasses are now very lightweight and portable, and can produce quantitative data with minimal set-up requirements or involvement of technicians. As a result, eye-tracking may be used in the future as an adjunctive assessment method during colonoscopy training. However, additional evaluation of other components of validity will be required before this is possible. Specific training in VGPs may also be worth incorporating into future training programmes.

Ethical approval

The study was performed with ethical approval from the West London & GTAC Research Ethics Committee on 5th August 2014, reference 13/LO/0119.

Disclosure statement

No conflicts of interest.

SE, JH and MHS conceived the study.

UK, KEH, JH collected data

UK, SE, JH, MHS analysed and interpreted the data.

AD provided guidance on the study design and analysis.

UK, SE, JH, MHS drafted the manuscript.

All the authors read, edited, and approved the final version of the manuscript.

All authors had full access to all the data in the study.

Data availability statement

All data generated or analysed during this study are included in this article. Further enquiries can be directed to the corresponding author.

Additional information

Funding

This work was supported by a grant from the Royal College of Surgeons (Eng.). The authors have read the journal’s guidelines on finances and have no further conflicts of interest to disclose.