Objective assessment for open surgical suturing training by finger tracking can discriminate novices from experts

ABSTRACT

It is difficult, time consuming and expensive to assess manual skills in open surgery. The aim of this study is to investigate the construct validity of a low-cost, easily accessible tracking technique for basic open suturing tasks. Medical master students, surgical residents, and surgeons at the Radboud University Medical Center were recruited between September 2020 until September 2021. The participants were divided, according to experience, in a novice group (≤10 sutures performed) and an expert group (>50 sutures performed). For objective tracking, a tablet with SurgTrac software was used, which tracked a blue and a red tag placed on respectively their left and right index finger. The participants executed four basic tasks on a suturing model: 1) knot tying by hand, 2) transcutaneous suturing with an instrument knot, 3) ‘Donati’ (vertical mattress suture) with an instrument knot and 4) continuous intracutaneous suturing without a knot. In total 76 participants were included: 57 novices and 19 experts. All four tasks showed significant differences between the novice group and expert group for the parameters time (p<0.001), distance (p<0.001 for Task 1, 2 and 3 and p=0.034 for Task 4) and smoothness (p<0.001). Additionally, Task 3 showed a significant difference for the parameter handedness (p=0.006) and Task 4 for speed (p=0.033). Tracking index finger movements using SurgTrac software on a tablet while executing basic open suturing skills on a simulator shows excellent construct validity for time, distance and motion smoothness in all four suturing tasks.

KEYWORDS:

• Tracking
• surgical simulation
• basic suturing tasks
• open surgery
• construct validity

Introduction

It is important to have effective surgical training devices that can be used regularly to maintain surgical skills. Tools to train surgical skills are investigated and developed frequently [1–9]. Most of these training tools are used to train minimal invasive skills [1–4,6,7]. For example, box trainers or virtual reality simulators are developed, which can be used to track laparoscopic instruments and some even provide the trainee with haptic feedback [1,10,11]. New and more advanced techniques to practice minimal invasive surgical skills continue to develop, while devices to adequately train and assess open surgical skills fall behind. Several tools to enable the practice of open surgical procedures do exist, ranging from basic suturing pads to three-dimensional, layered models and even (live) animals [9,12–16]. For all these tools, an expert is necessary to provide the trainee with feedback, which make their use less flexible. This factor makes this approach unsuitable for self-directed continuous training. Furthermore, those experts use observer-based tools, such as OSATS [17], GERT [18] or UWOMSA [19], which are always more or less subjective. For adequate assessment a training tool with objective parameters is desired. An available technique with objective parameters that overcomes the need for expert observation, uses electromagnetic motion tracking to track hand movements during an open surgical procedure [8,20]. Unfortunately, this system is very expensive and consequently not fitted for broad implementation in continuous training.

In previous research, we showed that a low-cost tracking tool has potential to be used in open surgical simulated tasks for the tracking of finger movements [21]. This technique has the potential to overcome the need for expert observation, while remaining cost-effective. Therefore, this assessment tool could be useful in the training of open surgical skills. This tracking software has been used in MIS training and has shown be able to discriminate between expertise levels (construct validity), however this has not been evaluated in the use for open surgical training. Prior to using this system as a training device, it is important to demonstrate a good construct validity. The aim of this study is to investigate the ability to discriminate between expertise levels of this low-cost, broadly accessible technique of finger tracking in simple open surgical suturing tasks.

Methods

Participants

Surgeons, surgical residents and medical students at the Radboud University Medical Center, Nijmegen, The Netherlands, were recruited to participate in this study. Participants were included from September 2020 until September 2021. The medical students were included during the first week of their surgical rotation (internship) and assumed to be novices in surgical suturing skills, because they had no previous surgical rotation. The students had been taught basic suturing skills during education prior to their surgical internship, after which they were included, before any clinical exposure. Therefore, they had knowledge on how to perform the sutures needed for this study and did not need additional guidance during the study. Surgical residents and surgeons were perceived as suturing experts for the selected tasks. First, the participants completed an informed consent form and a short questionnaire, regarding their suturing experience. Novices were only included if they had no previous experience prior to their suturing education and experts had equal to or more than fifty sutures as previous experience. This was based on previous studies stating that expertise should be reached by 50 repetitions for most basic surgical procedures, such as laparoscopic cholecystectomy [22–24]. By ensuring that the experts had done at least 50 repetitions on all different suturing skills that we evaluated in this study, we regard them as experienced.

Participants agreed with anonymous processing of the collected data. A waiver for medical ethical approval was provided, because of the non-medical intervention setup of this study.

Equipment To objectively track participant’s finger movements, a tablet with tracking software and an open surgical simulator were used. Figure 1a,b show a setup of the used materials. To track the participants’ finger movements, SurgTrac software (eoSurgical Ltd., Edinburgh, United Kingdom) was used [25] as a tablet application. The tablet was placed in a stand above the right shoulder of the participant. The distance between the camera and the simulator was set at sixty centimeters to have an adequate overview. SurgTrac software has been developed to track minimally invasive surgical instruments, tagged with a blue and red sticker, in a simulated setting. The software recorded 30 frames per second for accurate constant motion tracking [7]. Recent research has shown that it is possible to use SurgTrac software for finger tracking in open surgical simulation [21]. Participants wore white surgical gloves with the right index finger tagged by a red balloon-tube and the left index finger tagged by a blue balloon-tube. A model by PediatrickBoxx (Nijmegen, The Netherlands) [26] was used for a standardized simulation task of open surgery. The model consists of a wooden cast placed in a position of forty-five degrees angle to the table, with a suturing pad by EduStitch [27]. This achieves an adequate view of the task execution through the camera.

Figure 1. a and b Research setup with a Lenovo P10 tablet in a stand and a simulator by PediatrickBoxx.

Tasks

All participants executed four suturing tasks on the given equipment in the following order:

1. Knot tying by hand (Figure 2a): participants tied a reef knot consisting of an underhand and overhand throw.

   Figure 2. a) Task 1: Knot tying by hand. b) Task 2: Transcutaneous suture. c) Task 3: ‘Donati’ suture. d) Task 4: Intracutaneous suture.

   

2. Transcutaneous suturing and knot tying with instruments (Figure 2b): participants executed one transcutaneous suture on the incision of their pad and tied the suture using their instruments.

3. Vertical mattress suturing (‘Donati’ suture) and knot tying with instruments (Figure 2c): participants executed one vertical mattress suture on the incision of their pad and tied the suture using their instruments.

4. Continuous intracutaneous suturing without knot tying (Figure 2d): an intracutaneous knot was tied in advance by the researcher at the upper side of the incision. Participants made an intracutaneous suture through the total incision in their pad (four centimeters in length). No intracutaneous or extracutaneous knot was done at the end of the suture

Outcomes

The SurgTrac software on the tablet tracked the red and blue tag on the index fingers of the participants during the execution of the suturing tasks. The parameters that the SurgTrac software measures are time of executing a task (in seconds), distance traveled by the left and right tag (in meters), distance between hands (average distance between the red and blue tag in centimeters), hands off-screen (in percentage of time), speed (mean speed of right or left hand, in millimeters/second), acceleration (mean acceleration of right or left hand, in millimeters/second [2], smoothness (mean smoothness of motion of right or left hand, in millimeters/second [3] and handedness (percentage right- and left-hand usage). These parameters, as measured by SurgTrac have been frequently validated for training of minimally invasive skills [7,28,29] especially time and distance. Because no other data for open surgery and finger-tracking is available for this software, the proven valid parameters in MIS skills, namely time and distance serve as the primary outcome parameters for the validation of finger-tracking in this open surgical suturing tasks study.

The other parameters provided by the software will serve as secondary outcome parameters. All parameters will be included in the construct validation process, barring the parameter off-screen, because it lacks clinical relevance in open surgery (no use of a screen).

The parameters are measured separately for the red and blue tag by SurgTrac. The combined total score of the red and blue tag for the parameter distance, consisted of distance travelled by a participant’s right and left hand in total. The combined total scores of the parameters speed, acceleration and smoothness were analysed as a mean score for right and left hand in total. The score of handedness is described as difference between percentage usage of the right hand and percentage usage of the left hand.

Statistical analyses

Data was analyzed by IBM Statistical Package for Social Sciences (SPSS), version 25. A p-value of<0.05 was assumed statistically significant. To compare outcomes of the novices and expert group, Mann Whitney U tests were used in case of data with non-normal distribution. All other data were represented as descriptive statistics. For a desired power of 0.80 with a power rate of 0.05 a sample size of 19 participants is required. This was based on the expected and clinically relevant differences in time to complete the task. For basic open suturing tasks we assumed 45 seconds as a clinically relevant difference. However, a large variation in results was expected in the novices group. Therefore, more novices were included, to overcome this problem and strengthen the data.

Results

Demographics

76 participants were included, 57 participants in the novice group and 19 participants in the expert group. The novice group consisted of medical students without any suturing experience, prior to their basic training, just before this study. The expert group comprised of six residents and thirteen surgeons, with varying, but at least three years of surgical experience. All the experts had executed fifty or more sutures in their medical career.

The mean age was 23.8 years (SD 3.0) in the novice group, 43.9 years (SD 9.9) in the expert group and 28.9 years (SD 10.3) overall. The novice group consisted of fourteen male and forty-three female, where the expert group consisted of more male than female, namely fifteen male and four female participants. No significant differences in outcome parameters were found for gender within both groups.

Construct validity

Table 1 shows the primary outcome parameters of the four executed tasks in both groups. Novices and experts differed significantly for the parameters time (p < 0.001) and distance (<0.001<p < 0.041) in all four tasks. The differences between novices and experts for the primary outcome parameters are visualized in Figure 3a,b. In Table 2 the secondary outcome parameters are shown. Of those, smoothness differed significantly (p < 0.001) for all four tasks. Furthermore, for Task 2 and 3 there was a significant difference between novices and experts in handedness (0.001<p < 0.046). While Task 4 shows a significant difference in distance between the hands (p = 0.015), in addition to the significant differences in time, distance and smoothness.

Figure 3. a and b Bar graph of primary outcome parameters time and distance.

Table 1. Primary outcome parameters of novices and experts. Mann Whitney U tests. P-value<0.05 is assumed as a statistically significant difference.

Primary outcome parameters		Task 1 (Median, IQR)	Task 2 (Median, IQR)	Task 3 (Median, IQR)	Task 4 (Median, IQR)
Time (s)	Novices	41.0 (50.5)	108.0 (49.0)	173.0 (72.0)	294.0 (116.0)
	Experts	9.0 (6.0)	27.0 (11.0)	43.0 (8.0)	110.0 (54.0)
	P-value	<0.001	<0.001	<0.001	<0.001
Distance (m)	Novices	2.2 (6.3)	7.8 (8.4)	13.4 (17.2)	27.3 (27.7)
	Experts	0.8 (1.9)	1.7 (2.6)	2.8 (4.6)	12.4 (26.4)
	P-value	<0.001	<0.001	<0.001	0.041

Table 2. Secondary outcome parameters of novices and experts. Mann Whitney U tests. P-value<0.05 is assumed as a statistically significant difference.

Secondary outcome parameters		Task 1 (Median, IQR)	Task 2 (Median, IQR)	Task 3 (Median, IQR)	Task 4 (Median, IQR)
Handedness (%)	Novices	60.0 (57.2)	33.1 (74.8)	73.3 (27.7)	71.8 (24.6)
	Experts	71.1 (71.4)	69.5 (38.6)	44.5 (56.0)	57.6 (36.9)
	P-value	0.606	0.046	0.001	0.063
Speed (mm/s)	Novices	16.6 (29.8)	20.1 (23.2)	22.3 (23.4)	26.7 (25.6)
	Experts	18.7 (51.9)	23.5 (38.7)	21.9 (35.0)	34.6 (51.9)
	P-value	0.185	0.092	0.305	0.051
Acceleration (mm/s2)	Novices	3.4 (2.0)	3.2 (1.2)	3.7 (1.5)	3.5 (1.5)
	Experts	2.5 (3.4)	4.3 (2.7)	3.1 (1.4)	2.9 (1.2)
	P-value	0.487	0.288	0.099	0.062
Smoothness (mm/s3)	Novices	0.7 (0.9)	0.3 (0.2)	0.2 (0.1)	0.1 (0.1)
	Experts	3.7 (6.1)	1.3 (1.0)	0.7 (0.5)	0.3 (0.1)
	P-value	<0.001	<0.001	<0.001	<0.001
Distance between (cm)	Novices	2.8 (2.8)	3.4 (1.4)	3.4 (1.4)	3.6 (1.5)
	Experts	5.8 (7.1)	3.6 (4.5)	3.8 (3.5)	5.4 (4.8)
	P-value	0.111	0.806	0.710	0.015

Discussion

The SurgTrac software, used in this study for tracking index finger movements, was initially validated for tracking laparoscopic instruments during execution of minimally invasive surgical tasks [4,25]. In a prior experimental study (accepted manuscript) the feasibility of tracking index finger movements using SurgTrac was confirmed. Therefore, this study focusses on the construct validation of this assessment method for open surgical training, to evaluate the ability of this assessment method to discriminate between novices and experts.

All executed basic suturing tasks in this study showed a significant difference in the outcome parameters time, distance and smoothness when comparing novices to experts. For these parameters construct validity is established. For the other parameters measured by SurgTrac, such as handedness and distance between hands, construct validity could not be established. Although these latter two do not seem to have any clinical relevance in the assessment of the trainee’s skill level. Unequal distribution of using the right and left hand or the distance between both hands, does for itself not affect the quality of a suture and has no clinical relevance in the assessment of it. Smoothness, on the contrary, is an indicator of a participant’s instrument handling and a relevant indicator for the executioner’s level of basic suturing skills. Because the parameter smoothness is calculated using the parameters time and distance, it is debatable whether this parameter as such can be best used to differentiate between skills levels of the trainees. Nevertheless, this calculated parameter did show a significant difference between novices and experts, which means it can potentially be used by a trainee to monitor their own skills level. The current simulation and assessment setting with tracking finger movements can adequately discriminate between a novice or expert level for basic suturing skills. Tracking finger movements using SurgTrac while performing basic suturing skills on a simulator is therefore a promising setting to train and objectively assess basic open suturing skills. The aim of this study is to evaluate whether and which parameters of the SurgTrac application are able to discriminate novices from experts. This had been done in numerous studies as a construct validity study [30–37]. We acknowledge that this is a basic form of validity testing, however, it is a first step towards the assessment of the true potential and capabilities of this tracking software in the use for open surgical training. The next step is a concurrent validity study, which we are currently doing, based on the results that we have from this study. After that a pass-fail cut of should be established, to be able to truly assess and inform the trainee whether they are proficient or not.

The previously mentioned simulation setting is, unlike previous investigated settings, such as using sensory gloves with electromagnets [8,20], easily accessible and relatively affordable, because of the use of simple materials like gloves and balloon-tubes. Consequently, this setting is accessible for nearly every potential trainee in virtually every country, because the SurgTrac application can be downloaded to any smartphone or tablet. This improves the possibilities of medical students, residents and doctors to train and assess their skills independently. Furthermore, this tool is very compact, which makes it usable in every desired place, like home or work. Another advantage of this tool can be that no expert is needed for assessment, allowing a trainee to train at any suited time.

A simulator for basic open suturing tasks combined with finger motion tracking, provides an easily accessible simulation setting, which is available for everyone to train anytime. This is not only to learn new surgical skills, but also to maintain the optimal level of skills, especially if there is no consistent exposure in the clinical setting. Because continuous training is a necessity to maintain surgical skills, as shown before in MIS studies on this subject [29,38–41].

This study showed that time, distance and smoothness are valid parameters to assess technical aspects of performing a suture in open surgery. Yet, these parameters, cannot be extrapolated to assess quality aspects, such as firmness of the knot, distance between sutures or distance between the wound edges. Clinical outcome of any surgical task is the ultimate outcome parameter, therefore further research should focus on correlating clinically relevant parameters with quality of technical performance.

Limitations

No information was collected about the right- or left-handedness of participants. Analyses were performed using the combined total scores of the trainees right and left hand, in which dominance had no influence on the outcome parameters. Nevertheless, it would be interesting to further explore the differences between novices and experts taken hand dominance into account and analyse their dominant and non-dominant hand separately. Although not all parameters provided by the SurgTrac software seemed to be relevant for open surgery, these were all included in the study to avoid assumption bias. Due to the position of the tablet, and thus camera, with the tracking app, sometimes the fingers were not clearly in view, which could affect the outcome data. This was visible when looking at the out of view data and when this was too much, it was clear that the data could be less reliable. However, this did not seem to affect our results. When further validation studies have been performed, such as concurrent validity, where the results of this assessment method are compared to expert assessment, for example, the true potential will be evident. Based on this kind of studies a pass-fail score could be made, which trainees can use to evaluate their skills independently. However, it could also state that more parameters are needed for a true relevant assessment, which are not accounted for in this method.

Conclusion

Tracking index finger movements using SurgTrac software on a tablet, while executing basic suturing skills on a low-cost surgical suturing model, shows excellent construct validity for time, distance and motion smoothness in all four suturing tasks. This new open surgical assessment method can be implemented in any training setting, because it is easily set up as an application on any mobile device, making it a potent objective assessment tool in open surgical training.

Availability of data

The datasets used and analysed during the current study are available from the corresponding author on reasonable request.

Acknowledgments

No further acknowledgements need to be appointed.

Disclosure statement

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. No author disclosures or conflicts of interests have to be disclosed, other than the fact that Dr. S. Botden is co-founder of PediatrickBoxx, a small non-for-profit cooperation, that has developed the wooden casing used in this study. The minimal profit on selling the models is used in the development and production of new models.

Additional information

Funding

The author(s) reported there is no funding associated with the work featured in this article.