Endourological simulator performance in female but not male medical students predicts written examination results in basic surgery

Abstract

Objective. The objective of this study was to examine the correlation between endourological simulator performance and demonstrated theoretical knowledge in the basic surgical sciences. Material and methods. In total, 158 fourth year medical students participated in the study, 83 females and 75 males, all surgical novices. All students performed the flexible endoscopic task Hall of Fame in the urological simulator URO Mentor^™. Later during the same semester all students took the final theoretical examination in surgery. Results. In female medical students a significant correlation was found between surgical simulator performance and the examination results (r = –0.22, p = 0.04). There was no statistically significant correlation when looking at the total study population (r = –0.04, p = 0.58) or when looking at male medical students (r = 0.01, p = 0.9). Conclusion. Female medical students completing an endourological simulator task more efficiently passed the theoretical examination in the basic surgical sciences with significantly higher scores than females with low efficiency in the urological simulator. There are likely to be several explanations for this correlation, such as motivation and a lower amount of current video gaming experience.

Key Words::

• medical students
• surgical simulation
• surgical training
• urology

Introduction

Simulator training is becoming increasingly important in graduate and specialist clinical education. Society, as well as healthcare management organizations, is highly interested in efficient and low-risk healthcare, leading to increased demands on simulated training of healthcare professionals in order to increase the efficiency of patient care without risking an increased number of medical errors. The development of minimally invasive surgery within the urological speciality is no exception from surgical procedures in general, regarding the need for new and sophisticated valid training methods in safe and realistic settings. Arora et al. recently proposed a framework for the incorporation of simulation into specialist training in urology [1].

Simulator training has been the subject of intensive research during the past decade. Not surprisingly, research has shown a transfer effect from surgical simulator training to real surgical tasks, and that it may be an effective way of acquiring certain technical skills in surgery in general [2–5] and urology in particular [6]. Simulators have been used to test different training programmes as well as to examine the abilities needed for minimally invasive surgery. Basic research within the field has shown that working memory, the ability to retain information during a delay and act upon that information [7], and visual spatial ability, used when mentally manipulating and rotating three-dimensional figures [8], are some of the abilities of importance for the prediction of simulator performance. Furthermore, it has been shown that training of cognitive skills significantly improved the performance on an endovascular simulator [9]. Systematic video game training for 5 weeks has also been shown to improve simulator performance in a group of surgical novices [10]. Since many studies have shown that psychomotor ability is an important predictor of both learning rate and performance for complex laparoscopic tasks, questions have been raised as to whether we should rely on markers such as academic record instead of psychomotor skills when recruiting future surgeons [11,12]. The role of theoretical knowledge in surgical performance is not known. Further work is required to develop a valid taxonomy of individual surgeons' non-technical skills for training and feedback [13].

Used properly, high-fidelity medical simulations facilitate learning in medical students [14]. However, there is a lack of data on the importance of advanced image-guided simulations within the curriculum of undergraduate studies in basic surgery and urology. What effects could training in a urological simulator (or other surgical simulator) have on undergraduate students' knowledge acquisition in the basic surgical sciences? So far, it has not been investigated whether their theoretical knowledge is associated with surgical simulator performance.

The aim of this study was to analyse the correlation between demonstrated performance in a surgical simulator and the acquisition of theoretical knowledge.

Material and methods

All 166 medical students were tested in the validated URO Mentor^™ (Simbionix, Cleveland, OH, USA) endourological simulator during the rotation in urology [14–16]. This group of randomly chosen students from the course in basic surgery was assigned by the teacher to perform the task Hall of Fame in the URO Mentor. Eight students were excluded owing to non-completion of the task. Hence, 158 students remained in the study: 83 females and 75 males. None of the students reported any previous experience in virtual reality image-guided surgical simulators. The study population was recruited from the regular fourth year basic course in surgery at the Medical School, Karolinska Institutet. All enrolled students participated after approval on a voluntary basis. Information was gathered between January 2004 and May 2010. The study was approved by the local ethics committee.

Every student was given a standardized oral instruction regarding the goals of the task to be performed in the simulator by the same instructor during the entire study. All students were given a training period of approximately 20 min in the simulator. In the task Hall of Fame, the task was to navigate in a virtual gym hall with connected corridors resembling the anatomy of the real urinary tract, using a flexible ureteroscope. During the procedure they were supposed to collect basketballs with an inserted basket and shoot them through a hoop. Performance was measured by the total time finding and scoring all 13 basketballs through the hoop. Eight students did not complete the task and were therefore excluded from analyses. The students were given the instruction to find and score the basketballs as quickly as possible.

Between 4 and 12 weeks after the simulator test occasion the students took a final written examination in basic surgery, consisting of questions regarding urology, anaesthesiology, orthopaedics, radiology and general surgery. The curriculum and learning objectives were identical during the study period; however, the questions in the examination did differ in minor parts from year to year, although the proportion of the different topics remained the same.

After the examination, the total percentage of correct answers from the test was correlated with the total time in seconds in the urological endoscopic simulator. To evaluate hypotheses of variables in contingency tables, the chi-squared test was used or, in the case of small expected frequencies, Fisher's Exact Test. Statistical comparisons to test differences between two independent groups were made using the Student's t test for uncorrelated means, after validation for normal distribution by the Shapiro–Wilk test. The Pearson correlation coefficient was used to test independence between variables. In addition, descriptive statistics were used to characterize the data. All analyses were carried using the SAS system for Windows 9.2 (SAS Institute, Cary, NC, USA) and 5% levels of significance were considered. In the case of a statistically significant result the probability value (p value) has been given.

Results

Age, simulator performance and examination results from the total study population (for the whole group as well for male and female subjects) are summarized in Tables I, II, III. There was a statistically significant difference in the task Hall of Fame between male and female students in total time (p < 0.0001, –222.9). There were no statistically significant differences regarding age or theoretical examination results.

Table I. Age of the study population.

Group	Mean	SD	Lower quartile	Upper quartile
All (n = 158)	26.6	4.1	24	28
Male (n = 75)	26.9	4.1	24	29
Female (n = 83)	26.4	4.1	24	27

Table II. Test percentage correct answers in the final theoretical examination in basic surgery.

Group	Mean	SD	Lower quartile	Upper quartile
All (n = 158)	77	6.5	74	81.1
Male (n = 75)	76.1	6.5	73	80
Female (n = 83)	77.8	6.3	74	82

Table III. Total time in simulator exercise (in seconds).

Group	Mean	SD	Lower quartile	Upper quartile
All (n = 158)	753.2	252	542	951
Male (n = 75)	636.1	208.9	463	761
Female (n = 83)	859	241.6	677	1033

There was no statistically significant correlation between surgical simulator performance and the theoretical examination when looking at the total study population (r = –0.04, p = 0.58) or when looking at male medical students (r = 0.01, p = 0.9). However, in female medical students there was a significant correlation (r = –0.22, p = 0.04). The correlation between the theoretical examination results and simulator performance when looking at females and males is illustrated in Figure 1(a) and (b), respectively.

Figure 1. Correlation between theoretical examination results and simulator performance: (a) female students; (b) male students.

When dividing all the students into quartiles according to simulator total time, where the first quartile corresponds to the fastest completion time, there were significant differences regarding total time in the first (p < 0.001, –153), second (p < 0.001, –228.8), the third (p < 0.001, –268.8) and fourth (p < 0.001, –231.8) quartiles; that is, male students were faster in all quartiles. Regarding test percentage, only in the first quartile there was a significant difference, where female students had better results than males (p = 0.03, –4.16).

Discussion

In this study, performance in the URO Mentor simulator predicted the level of theoretical knowledge in basic surgery in female but not in male medical students. Further experimental studies will test whether this statistically significant correlation is a causal link. The finding implies that an effort to increase female students' simulator performance may affect knowledge acquisition within the basic surgical sciences.

Male students had a faster completion time in the urological simulator task. This is very likely to be due to a combination of collaborative factors. One reason could be the males' extensive current video gaming experience compared to the female students [10]. With this high current gaming experience the attitude towards knowledge and practical skills could have been modified, e.g. less technical respect using the simulators. Previous studies have shown that video game experience correlates with surgical simulator performance and that it is actually possible to use systematic video game training to enhance simulator performance [10]. Male students performing worse on the theoretical examination 4–8 weeks after the simulator test performed on a par in the simulator with male students performing well or excellently on the theoretical test. This contrasts with the results achieved by the female students, where low theoretical test performers performed significantly worse in the simulator compared to the good/excellent females on the same test. Gender differences in attitudes towards an unknown new task, especially practical ones, could be the reason for this effect. One could assume that motivation drives the acquisition of surgical theoretical knowledge and practical skills. If that were the case, an explanation for the finding in female medical students could be that highly motivated students achieve higher scores on the surgical simulator test and also study more intensively for the theoretical examination. It is possible that a low performance score in the simulator affects the individual student in a negative way, inducing non-productive learning behaviour towards the examination in the basic surgical sciences. Although debatable, it is the authors' opinion that the use of advanced image-guided simulators in the undergraduate courses of the MD university programme safely adds important information to help students to understand the complex clinical picture of their studied disciplines. In a best evidence medical education (BEME) guide from 2005, it is stipulated that integration into the overall curriculum is one of the conditions facilitating learning among trainees [17]. Furthermore, feedback during the learning experience, repetitive practice with increasing levels of difficulty, clinical “variation” through the use of simulators and individualized learning in the simulator are other factors of importance for effective learning [17]. Recently, the course in basic surgery at Karolinska Institute at the Karolinska University Hospital in Huddinge has decided to implement increased simulator training programme during the course. The effects of this decision are about to be evaluated and the future will tell whether this will lead to better theoretical knowledge and higher motivation.

The simulator test in this study measured only total time and not other metrics such as trauma, smoothness and perioperative bleeding. It is therefore possible that females work more slowly but more carefully and with fewer risks. Possible effects of the simulator instructor/tutor's gender, instructional tone and pedagogical ability on the different students can only be speculated about; however, a strength of the study is that every student had the same instructor. A possible weakness of the study is that the time passing between the simulator testing and the final examination was quite long and varied between students. In general, students acquire a great part of their theoretical knowledge during the final weeks before examination.

In future studies it would be interesting to investigate the role of motivation in regard to theoretical knowledge and simulator performance, and to evaluate the increased simulator training programme at the Karolinska Institute.

Acknowledgements

This study was fully supported by unrestricted research grants from Karolinska Institutet.

Declaration of interest: Drs Schlickum, Felländer-Tsai, Hedman and Henningsohn have no conflicts of interest or financial ties to disclose.