Teaching and Assessment of Medical Students During Complex Multifactorial Team-Based Tasks: The “Virtual on Call” Case Study

Abstract

Abstract

Although considerable efforts have been made to incorporate simulation-based learning (SBL) in undergraduate medical education, to date, most of the medical school curricula still focus on pure knowledge or individual assessment of objective structured clinical examination skills (OSCE). To this end, we designed a case study named “iG4 (integrated generation 4) virtual on-call (iVOC)”. We aimed to simulate an on-call shift in a high-fidelity virtual hospital setting in order to assess delegates’ team-based performance on tasks related to patient handovers (prioritisation, team allocation).

Methods

A total of 41 clinical year medical students were split into 3 cohorts, each of which included 3 groups of 4 or 5 people. The groups consisted of a structured mix of educational and cultural backgrounds of students to achieve homogeneity. Each performing group received the handover for 5 patients in the virtual hospital and had to identify and deal with the acutely unwell ones within 15 minutes. We used TEAM^TM tool to assess team-based performances.

Results

The mean handover performance was 5.44/10 ± 2.24 which was the lowest across any performance marker. The overall global performance across any team was 6.64/10 ± 2.11. The first rotating team’s global performance for each cycle was 6.44/10 ± 2.01, for the second 7.89/10 ± 2.09 and for the third 6.78/10 ± 1.64 (p = 0.099 between groups).

Conclusion

This is one of the first reported, high-fidelity, globally reproducible SBL settings to assess the capacity of students to work as part of a multinational team, highlighting several aspects that need to be addressed during undergraduate studies. Medical schools should consider similar efforts with the aim to incorporate assessment frameworks for individual performances of students as part of a team, which can be a stepping-stone for enhancing safety in clinical practice.

Keywords:

• virtual on-call
• simulation based learning
• team-based assessment
• high fidelity simulation
• handover

Modern medical school curricula stress the importance of improving communication and emphasise the acquisition of non-technical skills. Despite a considerably increasing trend of published studies on non-technical skills related to teaching for undergraduate students,¹ some medical schools still lag behind formal assessments of non-technical skills of their stakeholders.²

Designing teaching interventions that combine non-technical skills with applied medical knowledge elements have been facilitated by several Simulation-Based Learning (SBL) models.¹ Although considerable efforts have been made to incorporate SBL in the undergraduate medical education, to date, most of the medical school assessments focus on pure knowledge (exams) or individual assessment of objective structured clinical examination skills (OSCE).¹ Incorporation of assessments of medical students’ performance as part of a team is yet to be formally introduced in most medical school curricula. This could potentially be delivered in a form of team-based OSCE assessments ^2,3 using validated tools.⁴

Being a safe and efficient junior doctor can be challenging, and often necessitates the application of a melange of non-technical and technical skills with applied knowledge.^1,5 Junior doctors on call is a classic example where an individual (doctor) must work as part of a wider multidisciplinary team (MDT) under stressful circumstances.⁶ This requires the application of complex non-technical skills which can include closed loop communication with other junior or senior doctors, leadership, task-based focus with effective prioritisation, situational awareness etc.² One of the most critical steps in this chain is the effective delivery of multidisciplinary handover which is critical to maintain patient safety. During the handover process, junior doctors identify the applicable tasks to be done, distribute them across the team and prioritise them in a timely manner. Based on that, they apply their clinical and managerial knowledge to accomplish a huge variety of tasks and deliver safe care for their patients. Increasing workload, communication gaps, lack of staffing, diversities in both educational and cultural backgrounds of the staff, as well as stress during a busy shift can affect overall healthcare provision. Hence, many final year medical students complain that they do not feel confident or prepared for on call shifts and feel nervous during them.

To this end, we designed an innovative study named “iG4 (integrated generation 4 ^7–9) virtual on-call (iVOC)”. Using high fidelity SBL facilities of a virtual hospital environment, the iVOC study aspired to simulate a real on-call shift. Participants were clinical year medical students from different educational and cultural backgrounds. We primarily aimed to design a reproducible virtual on call model based on SBL. Our secondary aims were to measure team-based performance in order to gauge the ability of clinical-year medical students to prioritise the tasks generated after handover, including patient reviews according to clinical urgency.

Methods

Setting (esmsc.gr Course Concept)

Essential Skills in the Management of Surgical Cases – ESMSC Marathon Course ¹⁰ is a 3-day international multifaceted surgical course aimed at the undergraduate level. ESMSC combines a mix of technical and non-technical skills modules with basic and applied surgical science workshops. Technical skills modules involve high fidelity in vivo swine model simulation (anaesthetised pig), as well as lower fidelity ex vivo or dry lab modules. Basic Science workshops involve essential elements that the future surgeon should be proficient at, i.e. fluid resuscitation or Arterial Blood Gases Interpretation. Applied Surgical Science modules involve small group teaching in a case-based setting. Non-technical skills settings vary from low fidelity Simulation-Based Learning (SBL) to more advanced group-based high fidelity human factors scenarios.^2,3 The course concept and curriculum have been described in the literature previously;^5,10 the course concept has evolved throughout time using certain evidence and expert experience⁷ to facilitate the vision of holistic education.^8,9,11 Apart from the non-technical skills core, none of the other aforementioned modules are directly related to this current study. iVOC is a pilot novel study, which aims to expand the ESMSC curriculum in the direction of team-based non-technical skills education.

Setting (Facilities)

This is a case study which ran at the Virtual Hospital Setting of the Centre of Advanced Simulation and Education (CASE, Acibadem University Istanbul, Turkey) which offers a fully equipped hospital allowing high fidelity simulation. The unique advantage of this setting is that it allows simultaneous SBL scenarios to run, using advanced technology, achieving the highest possible fidelity. The rest of the ESMSC modules (unrelated to this study), took place at the in vivo and dry lab simulation facility of the same lab. We followed the standard operating procedures (SOP) of the CASE lab regarding the SBL standards of this study.

Participants (Selection of Delegates and Background)

ESMSC invites applications from several universities of the UK and the EU as well as other global medical schools including those in Turkey and India. Delegates are selected via a competitive process which includes submission of their CV and statement of interest. Each series of the course is attended on average by 40 delegates. Delegates should at least be at their clinical years of training and beyond or equivalent.

Patient and Public Involvement

This study involves students and not patients, hence there is no direct patient or public involvement.

Intervention (The “Virtual on Call - iVOC” Study Design)

The course participants were split in 9 groups of 4 or 5 people. We mixed students from different years of studies (experience) and educational background (diversity) to achieve equality (knowledge, clinical experience) across different groups and for it to be representative of an international healthcare service system such as the National Health Service (NHS) in the UK. For this study, we matched 3 groups in each cohort and ran 3 cycles of scenarios. Hence, we had 3 cohorts consisting of 3 groups and each cohort attended 3 cycles of scenarios. During each cycle, all teams performed once and were observed twice; this means 1 group was performing, and 2 groups were observing through 9 cameras in a special control room designed for this purpose (Figure 1).

Figure 1 Study map.

Scenarios

Each performing group was given a handover sheet which consisted of 5 scenarios (Appendix 1); 2 non-urgent and 3 which required urgent attention. Each patient had a location in the virtual hospital. The students in each performing group were meant to interact and identify the 3 cases that required input and split their capacity to deal effectively and simultaneously with all emergencies within 15 minutes. The other 2 groups were able to observe the performing group only whilst in the simulation rooms, but not during the handover process.

Scenarios: Level of Difficulty

As stated, each cohort included 3 teams performing. For the first team that performed (1st cycle of the cohort) we designed relatively familiar or common scenarios. The second team performing had identical concept scenarios but slightly more demanding to manage (increasing difficulty). The last team would have observed the 2 previous groups, hence we decided to exponentially increase the level of difficulty in the scenarios. In total we had 3 cycles/cohort; each cycle was designed to be gradually more challenging to account for the experience that teams were gaining through observation.

Assessment Tool and Outcomes

We used the Team Emergency Assessment Measure (TEAM) to assess students during the handover and during each scenario. TEAM involves 11 domains and a global score (Table 1). TEAM is a validated tool which was not only deemed appropriate for the circumstances but is also accepted as the gold standard tool to assess team performance as stated in a recent systematic review.⁴ Students could ask for help from seniors or nurses through the internal communication system throughout the cycles. Outcomes were defined by the TEAM tool, and this mainly included individual performance in a team-based setting; for the purpose of simplicity, we focused on global scores only.

Table 1 Team Emergency Assessment Measure – TEAM™

TEAM™ Item
The team leader let the team know what was expected of them through direction and commands
The team leader maintained a global perspective
The team communicated effectively
The team worked together to complete the tasks in a timely manner
The team acted with composure and control
The team morale was positive
The team adapted to changing situations
The team monitored and reassessed the situation
The team anticipated potential actions
The team prioritised tasks
The team followed approved standards and guidelines
Global Score

Note: Reproduced from TEAM^™ https://medicalemergencyteam.com/what-is-the-team/. Cooper S. Cant R. Connell C, Sims L. Porter J. Symmons M. Nestel D. Liaw SY. (2016) Measuring teamwork performance: Validity testing of the Team Emergency Assessment Measure (TEAM) with clinical resuscitation teams. Resuscitation. 101; 97-101.

Assessors and Data Collection

Two dedicated assessors were allocated in each room (where the scenarios took place) and two during the handover observing how students identify and allocate workload. All assessors were independent, and scenarios were video recorded; a third independent assessor marked each of the cycles. Each assessor filled out the TEAM form with a free space for feedback and had a dedicated code so we could go back and discuss any discrepancies. Data were transferred in an Excel file, following harmonisation with video recording results.

Peer Assessment

During each cohort there were 3 rotating teams; one performing and 2 teams observing and collating feedback which was communicated during the debriefing session after each cycle (15 minutes). Feedback sessions were together with assessors who discussed the assessment and provided feedback in the presence of all 3 teams. Each team was performing once per cycle and observing twice. The handover process was the only part of the performance which was not observed by the other two teams. This was to ensure we captured a baseline performance of the delegates in allocating and organising their handover tasks.

Statistical Analysis

We used inferential univariate statistics to describe data on IBM SPSS for Macintosh version 26. We assessed normality of data using Shapiro–Wilk test. P-value less than 0.05 was considered as statistically significant.

Results

Sample and Demographics

A total of 41 delegates attended the course; 27 (65.9%) from UK Medical Schools (King’s College London and Imperial College London), 6 (14.6%) from the University of Athens, 5 (12.5%) from Acibadem University (Istanbul, Turkey) and 3 (7.3%) from Indian Medical Schools. Thirty-four (82.9%) belonged to the age group 20–24 and 7 (17.1) to age group 24–30. All students were in their clinical year rotations.

Performance Summary

Data summary is provided in Table 2 which summarises each TEAM item score across the different rotating teams.

Table 2 Global Scores for Each Rotating Team

TEAM™ Item	Overall Performance (Mean±SD) (Based on 9 Teams)	First Rotating Team Performance (Mean±SD) (Based on 3 Teams)	Second Rotating Team Performance (Mean±SD) (Based on 3 Teams)	Third Rotating Team Performance (Mean±SD) (Based on 3 Teams)	Overall Handover Performance (Mean±SD) (Based on 3 Teams)	P value (<0.05) (Between Groups Associations)
The team leader let the team know what was expected of them through direction and commands	2.56±1.08	2.00±1.22	3.22±.97	2.78±.44	2.22±1.20	0.063
The team leader maintained a global perspective	2.50±1.18	2.00±1.32	3.22±1.09	2.56±.53	2.22±1.39	0.137
The team communicated effectively	2.83±1.00	2.89±.78	3.22±1.09	3.00±.50	2.22±1.30	0.171
The team worked together to complete the tasks in a timely manner	2.81±.98	2.78±.83	3.22±1.09	2.89±.60	2.33±1.22	0.293
The team acted with composure and control	2.81±1.03	2.67±.87	3.33±1.00	3.11±.78	2.11±1.17	0.055
The team morale was positive	2.89±.90	2.88±.35	3.44±.73	3.00±.71	2.22±1.20	0.028
The team adapted to changing situations	2.69±.93	2.63±.74	3.22±.97	2.33±1.00	2.56±.88	0.217
The team monitored and reassessed the situation	2.82±.94	2.75±.71	3.22±1.09	2.88±.99	2.44±.88	0.378
The team anticipated potential actions	2.65±.92	2.38±.74	3.11±1.05	2.38±.74	2.67±1.00	0.303
The team prioritised tasks	2.94±.83	2.75±.71	3.63±.74	2.75±.71	2.67±.87	0.054
The team followed approved standards and guidelines	2.67±.96	2.50±.53	3.63±.52	2.38±.92	2.22±1.09	0.006
Global Score	6.64±2.11	6.44±2.01	7.89±2.09	6.78±1.64	5.44±2.24	0.099

Note: Adapted from TEAM^™ http://medicalemergencyteam.com/. Cooper S. Cant R. Connell C, Sims L. Porter J. Symmons M. Nestel D. Liaw SY. (2016) Measuring teamwork performance: Validity testing of the Team Emergency Assessment Measure (TEAM) with clinical resuscitation teams. Resuscitation. 101; 97-101.

Baseline Handover Performance

The mean global (team) score for handover performance was 5.44/10 ± 2.24; although overall lower than any other element of performance this did not reach a statistically significant level.

Baseline Performance/Performance Improvement Across Each Rotating Team

The overall global (team) score for baseline performance across any team was 6.64/10 ± 2.11. The overall global score for performance for the first rotating team of each cycle was 6.44/10 ± 2.01, for the second rotating team was 7.89/10 ± 2.09, and for the third rotating team 6.78/10 ± 1.64. Comparison across the rotating teams did not show any statistically significant difference (p = 0.099). We did not compare handover performance across rotating teams as this was the only process not observed by the other 2 teams during each cycle.

Discussion

What We Found

We found baseline handover performance to be poor; this has been the case for all the cohorts who had a similar baseline score. Handover performance was remarkably lower for all elements compared with the rest of team performance for each scenario. This potentially indicates a lack of hierarchical thinking and communication gaps which can be attributed to either language barrier and/or mix of educational backgrounds. Handover requires a mix of background clinical knowledge with non-technical skills, which results in essentially a more complex-to-teach process, but is still vital to maintain safety in clinical practice. Providing direct observation of every part of the cycle (apart from handover) resulted in an improved performance between the first and the second rotating team; however, this was not the case for the third rotating team whose performance was still better than the first one, but worse than the second. This can be attributed to the fact that the third cycle had a disproportionally difficult cohort of scenarios. Overall, there is improvement in performance via peer observation only, which did not reach statistically significant results because of sample limitations. From the TEAM items, we noticed a slightly lower performance trend on team leader skills (items 1 and 2). Observing the inter-item (TEAM tool) differences can essentially point out the direction of the teaching interventions needed to establish effective composite skills i.e. handover or teamwork, which are the foundation of safety in modern medicine.

Interpretation of Our Findings: The Role of Training Junior Doctors to Effectively Handover to Maximise Patients’ Safety During On-Call Shifts

There is patchy knowledge and understanding of handovers and relevant pitfalls that can lead to serious incidents.¹² Although it is evident that the root cause of most serious incidents are down to communication,¹³ it is more challenging to identify the stage of communication at which an error occurs, e.g. during handover.¹⁴ Recent research suggests that handover may in fact play a significant role in malpractice claims.¹⁵ However, there is a gap in the evidence for this, which may lead to handover and its critical role in providing safe clinical practice being overlooked.

Handover is the bridge of communication amongst allied healthcare providers. It is vital that it is performed with no error by avoiding lack of information and clarity, which can be hindered in such dynamic hospital settings. Unfortunately, there are studies showing that handover is performed poorly; this is associated with morbidity and mortality.¹⁶ This has sparked interest in the field of handover. There have been reports of studies, particularly questionnaires, on understanding the views and perceptions of trainees on the importance of handover.¹⁷ Some have made recommendations such as examining the approach to handover of healthcare professionals, with others identifying other potential ways to deliver handover such as effective electronic systems, devoted times for handover and writing good summaries for patients.^18,19 However, such research is focused more on the delivery of handover, i.e. symptomatic treatment of poor handover delivery, rather than attempting to explore the underlying pitfalls in handover and identifying ways to overcome this. Other studies where handover interventions were made and assessed for outcomes have mostly demonstrated poor methodology and inability to repeat them.²⁰

Nowadays, handover teaching in undergraduate medical education consists of revised templates and mnemonics to guide students.²¹ There are also simulation-based learning opportunities, however, these focus on single-patient based scenarios without the complex interplay of having multiple patients to hand over, and other factors such as the involvement of multidisciplinary team members from different backgrounds.²²

Strengths of Our Study Model

This is one of the first reported high-fidelity simulation-based learning studies to assess the capacity of students to work as a team. It has been performed in a centre of excellence (Centre for Advanced Simulation and Education) which provided a virtual hospital environment. This is a major strength, because this environment allows direct observation of how students from different educational backgrounds, whose native language may not be English and/or their degree may not be delivered in English, working together in a gold standard high fidelity setting. In simple terms, a virtual hospital, makes the environment real, allowing in-depth observation of the students’ performance.

At the same time, iVOC study provides a blueprint of developing and refining an easily reproducible setting, globally adaptable, as most interventions described in the study can be adapted in lower fidelity setting which are provided by most medical schools.

Finally, our students represented educational backgrounds of popular health systems including the NHS, Turkey (Acibadem University) as well as Cyprus, Greece and Asia (India).

Limitations

We recognise a series of limitations; although we used an international cohort, this is still a single centre study. Firstly, the number of participants is limited to reach statistically significant results, however the iVOC concept is clearly showing some useful trends in our results, enough to draw safe conclusions on our root cause analysis for students’ performance. Secondly, although the clinical scenarios were meticulously designed to reflect the study purpose, they were not validated formally. Finally, we did not introduce inter-observer variability measurements as this would complicate the analysis process in a small cohort of numbers.

Conclusions and Future Directions: Time to Re-Consider Objective Structured Clinical Examination

Our primary aim was to simulate an on-call shift using a high-fidelity virtual hospital setting and assess students’ team performance of tasks related to patient handovers (prioritisation, team allocation) during a busy handover. This would serve as a stepping-stone to develop in a high-fidelity setting, a reproducible OSCE module. This will introduce a ground-breaking shift in the mindset of “individual-based performance OSCE” to a “team-based performance”. Medical schools should consider standardisation of similar efforts to result in pilot team-based OSCEs. Those team-based OSCEs can be more representative of everyday practice and can potentially provide the students with skills that prepare them for safe practice, especially in multi-cultural healthcare systems. In two of our previously published studies ^2,3 we discussed some alternative low-cost models for how group-based OSCEs can be introduced as part of the medical school assessments.

Modern medicine is shifting continuously to teamwork-led practice, and this should also be reflected in undergraduate assessments. Such initiatives can provide a global assessment framework, which can be a stepping-stone for enhancing safety in clinical practice. It could also result in smooth integration of healthcare practitioners from abroad, who may be new or not familiar with the healthcare system that they plan to work in (for instance the NHS).

Place of Study

Center of Advanced Simulation and Education, Acibadem Mehmet Ali Aydinlar University, Istanbul, Turkey.

Ethical Approval

The ESMSC course is compatible with the current 3R principles for animal-model simulation (refinement, replacement, and reduction). Ethical approval was granted and met directive 63/2010, PD 56/April 2013, according to the local ethics committee. The license reference number is 4857/15-09-2017, MS, AP et al. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards. All the participants have signed a consent form to participate in the study.

Author Contributions

All authors made a significant contribution to the work reported, whether that is in the conception, study design, execution, acquisition of data, analysis and interpretation, or in all these areas; took part in drafting, revising or critically reviewing the article; gave final approval of the version to be published; have agreed on the journal to which the article has been submitted; and agree to be accountable for all aspects of the work.

Disclosure

Dr Aikaterini Dedeilia reports grants from World Health Organization – Health Workforce Department funding for research on the COVID-19 effects on health workforce education, outside the submitted work. The authors report no conflicts of interest in this work.