Pivot to online learning for adapting or continuing workplace-based clinical learning in medical education following the COVID-19 pandemic: A BEME systematic review: BEME Guide No. 70

Abstract

Background

The novel coronavirus disease was declared a pandemic in March 2020, which necessitated adaptations to medical education. This systematic review synthesises published reports of medical educational developments and innovations that pivot to online learning from workplace-based clinical learning in response to the pandemic. The objectives were to synthesise what adaptations/innovation were implemented (description), their impact (justification), and ‘how’ and ‘why’ these were selected (explanation and rationale).

Methods

The authors systematically searched four online databases up to December 21, 2020. Two authors independently screened titles, abstracts and full-texts, performed data extraction, and assessed the risk of bias. Our findings are reported in alignment with the STORIES (STructured apprOach to the Reporting in healthcare education of Evidence Synthesis) statement and BEME guidance.

Results

Fifty-five articles were included. Most were from North America (n = 40), and nearly 70% focused on undergraduate medical education (UGME). Key developments were rapid shifts from workplace-based learning to virtual spaces, including online electives, telesimulation, telehealth, radiology, and pathology image repositories, live-streaming or pre-recorded videos of surgical procedures, stepping up of medical students to support clinical services, remote adaptations for clinical visits, multidisciplinary team meetings and ward rounds. Challenges included lack of personal interactions, lack of standardised telemedicine curricula and need for faculty time, technical resources, and devices. Assessment of risk of bias revealed poor reporting of underpinning theory, resources, setting, educational methods, and content.

Conclusions

This review highlights the response of medical educators in deploying adaptations and innovations. Whilst few are new, the complexity, concomitant use of multiple methods and the specific pragmatic choices of educators offers useful insight to clinical teachers who wish to deploy such methods within their own practice. Future works that offer more specific details to allow replication and understanding of conceptual underpinnings are likely to justify an update to this review.

Keywords:

• COVID-19
• systematic review
• clinical learning
• online learning
• remote learning

Introduction

Practice points

• Adaptations to online learning described areas of excellent practice, including the use of cloud-based repositories to allow for collaborative clinical discussion and decision-making and the use of planned time for conversation within the online learning environment.

• Clinical consultations through telehealth methods used several technologies which facilitated interaction between learner and patient. Safeguarding the personal details of participating patients was specifically highlighted.

• Many of the innovations and adaptations could have broad applications after the pandemic, supporting new models of medical education which overcome practical and geographic barriers to maximise collaborative learning.

The global pandemic resulting from the novel coronavirus (SARS-CoV-2) has had a drastic impact on the delivery of healthcare and medical education across the globe. The requirement for the medical education community to ensure the safety of students, faculty and patients alike is driving much of the change in practice. Physical distancing, conservation of finite supplies of personal protective equipment (PPE), self-isolation, increased clinical workloads, and institution-specific stressors have created a mirage of challenges for faculty and trainees. Every learner and medical educator has been impacted in a unique way, leading to a heterogeneous and non-uniform response to continuing medical education through and beyond the COVID-19 pandemic. The uniqueness of this pandemic is that all those involved in medical education, regardless of how the learning was delivered previously, had to rapidly adjust their approaches to maintain educational progress and limit the ‘collateral damage’ of the numerous and necessary public protection measures.

Practice-based learning is the backbone of physician education. Wards, operating theatres, clinics, community services, and laboratories are where doctors in training and physicians learn, forge professional identities, and develop an orientation towards patient-focused care that shapes their practice (Dornan et al. 2015). The initial response by educators was invariably to terminate or modify medical student clinical placements to maximise the healthcare systems capacity to cope with increased clinical pressures, translate classroom teaching into remote and virtual formats, and provide select cohorts of undergraduate students with access to a provisional license to practice medicine in advance of their graduation date (DeWitt 2020). For instance, in the United Kingdom and Canada, there was a coordinated effort to mobilise medical students to work clinically as paid practitioners or volunteers (Baker et al. 2020). This approach was not commonplace globally, with the majority shifting towards learning through relatively unfamiliar and untested means within the online space (Gill et al. 2020).

The restrictions placed on the traditional medical education model of workplace-based learning necessitated medical educators to adapt and develop new modes of practice to ensure a modicum of continuity in training and attainment of core competencies. There have been numerous reports of strategies describing the adaptations and innovation of learning from the clinical workplace to allow students to continue their learning, including remotely delivered didactics, simulation, and telehealth consultations. It is without question that COVID-19 is transforming medical education. The axiom ‘the new normal’ is a phrase used to recognise the seismic change in practice, both within and beyond healthcare. It is inevitable that the necessary changes in the delivery of care, the transition towards virtual clinical encounters, disruption in the balance and inertia of clinical teams, and radical changes to hospital and primary care processes, are mandating equally novel and innovative approaches within medical education. Despite the many challenges, there have been opportunities for workplace-based learning to continue, highlighting the transformative utility of innovation, new approaches to blended learning, and the remote delivery of simulation and remote consultation.

This Best Evidence in Medical Education (BEME) review builds on two published reviews focused on the developments within medical education in response to the COVID-19 pandemic (Gordon et al. 2020; Daniel et al. 2021). The second of these was a scoping review, which mapped the broad array of developments, spanning 127 journal articles, to guide more in-depth systematic reviews. The adaptations of learning into the online space were identified as a major area requiring further attention, particularly given the pace and scale through which new research was being published. This current BEME review is just one of a triad of reviews focused on the pivot to online learning. The others studied the transition of traditionally classroom-based education into virtual formats, separated into postgraduate and undergraduate learner groups (Khamees et al. 2021; Stojan et al. 2021).

This current systematic review focuses on the adaptations and innovations from workplace-based clinical learning to online learning, deployed in response to the COVID-19 pandemic within undergraduate and postgraduate medical education (PGME). This review addresses the following:

1. What learning adaptations or innovations in undergraduate and postgraduate medical education have been explicitly deployed to adapt or continue workplace-based clinical learning? (i.e. description, or ‘what was done?’) (Cook et al. 2008)

2. What is the reported impact of these developments or innovations for the learners? (i.e. justification or ‘did it work?’)

3. How and why were these specific adaptations and innovations selected by the educators? (i.e., explanatory and rationale)

Materials and methods

Similar to our two previous reviews, this current review was conducted in a rapid timeframe, fifteen weeks from inception through to completion. Despite the rapidity through which the review was conducted, academic rigor and systematicity were not compromised at any stage of the review process. This review was not suited to one single research paradigm. Instead, both positivism (applying the principles of systematic reviewing) and constructivism (utilising qualitative synthesis methods) were embraced. A study protocol was completed a priori and uploaded into the study repository on the BEME website (Gordon et al. 2021). Our methodology and findings were reported in alignment with the STORIES (STructured apprOach to the Reporting in healthcare education of Evidence Synthesis) statement (Gordon and Gibbs 2014) and BEME guidance (Hammick et al. 2010).

Search strategy

The search strategy was developed with an expert librarian, using the accelerator Polyglot search translation tool (Clarke and Braun 2013). We conducted an electronic search within four databases (Pubmed, EMBASE, CINAHL and PsychInfo). To identify additional relevant articles, we conducted a manual electronic search of MedEdPublish, given the contribution this database made to the previous reviews. The full search strategy for each database can be found in Supplemental Appendix 1. PubMed was searched between August 2020 and 21 December 2020, to identify additional studies published after the initial rapid BEME review and its scoping update, which encompassed literature published between 1 January 2020–18 May 2020 and 1 May 2020–09 September 2020, respectively. The same search strategy was employed within this current systematic review as the development update BEME review (Daniel et al. 2021) and the other two BEME reviews on the pivot to online remote learning from the classroom (Khamees et al. 2021; Stojan et al. 2021).

After the search strategy was deployed, de-duplication was conducted using Endnote (Bramer et al. 2016). Retrieved citations were then uploaded into DistillerSR (Evidence Partners, Ottawa, Ontario Canada), a dedicated online management solution for performing systematic reviews. Further de-duplication was conducted within Distiller using the de-duplication function. Title and abstract screening were performed independently by two authors against an initial set of inclusion and exclusion criteria, which were used to identify all studies describing developments in medical education explicitly deployed in response to COVID-19. These inclusion and exclusion criteria were identical to those used within the previous two reviews (Gordon et al. 2020; Daniel et al. 2021). Inter-rater reliability was calculated using Cohen’s Kappa statistic (McHugh 2012). Full texts were retrieved and independently reviewed by two authors against inclusion and exclusion criteria, described below, to identify new studies describing online learning within medical education across both clinical contexts. Discrepancies at all stages were resolved through discussion, including a third author as required until consensus was reached.

To ensure we embraced the totality of literature, two authors independently assessed the 176 papers included across the first BEME review and its update (Gordon et al. 2020; Daniel et al. 2021), initially sieving out the studies which did not focus on remote or online learning, before assigning the remaining studies amongst the three individual categories: clinical pivot, classroom pivot (undergraduate medical education) and classroom pivot (postgraduate medical education).

Inclusion/exclusion criteria

The following inclusion criteria were used:

• Studies that described developments or innovations within medical education that were deployed explicitly in response to COVID-19.

• Studies that involved undergraduate medical students and/or postgraduate doctors.

• Studies that considered modified Kirkpatrick’s outcomes (level 1: satisfaction/reaction, level 2a: change in attitude, level 2b: change in knowledge or skill, level 3: behavioural change, level 4a: organisation practice change, level 4b: benefits to patients) (Kirkpatrick and Kirkpatrick 2016).

• Studies that described online educational development or interventions explicitly deployed as an innovation or adaptation to continue workplace-based (i.e. clinical) learning. This included developments or adaptations that:

  •  ○ Involved live patients or an aspect of the clinical workplace (e.g. ward rounds, multidisciplinary team meetings, virtual clinics, surgery).

  •  ○ Involved practical skill learning as an innovation or adaptation from learning that used to take place within the clinical workplace (e.g. surgical training and clinical skills education that was previously delivered using patients in the clinical workplace).

  •  ○ Involved the review of radiological images, histological specimens, or other clinical materials as an adaptation from reduced face-to-face clinical workplace exposure.

  •  ○ Described an adaptation or innovation from a clinical rotation or clinical experience that was disrupted following COVID-19.

   The following exclusion criteria were applied:

  • Opinion pieces, call for change, needs assessments, and other studies where no actual development has been deployed.

  • Studies that focused on developments or methods that were not explicitly deployed in response to COVID-19.

  • Studies that did not include undergraduate medical students or postgraduate doctors.

Data extraction

We modified the data extraction form used within the previous reviews to align with the aims of this review (Gordon et al. 2020; Daniel et al. 2021). The form was loaded into Google Sheets to allow synchronous review and sharing of extracted data. A team meeting was held before data extraction to ensure a shared understanding of terms and approach to enhance inter-rater reliability. Primary studies were assigned to author pairs for extraction, including those from the previous two reviews. At least one author within each pairing had participated in one or both of the previous reviews. Within pairs, extraction was performed independently prior to reconvening for corroboration. Any discrepancies were resolved through discussion or involvement of a third author (CGC).

The data extracted included the following:

• Article details (author(s) and month of publication)

  •  ○ Journal of publication

  •  ○ Length of publication (number of pages)

  •  ○ Article type (i.e. original research, letter to the editor)

• Context

  •  ○ Geographical origin of the development

  •  ○ Learner details (number of learners), level of medical education (i.e. undergraduate medical education, postgraduate medical education)

  •  ○ Medical specialty, if applicable

• Intervention deployment

  •  ○ Setting of the adaptation or innovation

  •  ○ Stated purpose of the deployment (i.e. what problem was addressed?)

  •  ○ What was the adaptation or innovation a replacement for? (i.e. what was delivered prior?)

  •  ○ Brief summary of the development

  •  ○ What resources were required (i.e. cost, time, material resource)

  •  ○ Theoretical models or conceptual frameworks explicitly described by the authors

• Intervention outcome(s) using the modified Kirkpatrick outcome level or similar (Kirkpatrick and Kirkpatrick 2016)

• Results

  •  ○ Summary of results

  •  ○ Lessons learnt

  •  ○ Summary of conclusions

• Risk of bias assessment

Quality assessment

Clinical education research includes a broad range of methodological disciplines, including medicine and social science. Reviews within these fields potentially include studies utilising a range of methods, both quantitative and qualitative. Quality assessment is therefore challenging and needs to be flexible to adapt to the heterogenous research questions, methods, and contexts. As per the first COVID-19 BEME review (Gordon et al. 2020), we again considered quality assessment as two distinct elements:

1. The risk of bias or quality of the study methodology

2. The risk of bias or quality of study reporting

The second point is critically important because only when the development is robustly and transparently described, can educators and researchers hope to replicate the results in other contexts. This approach to risk of bias has been used within both BEME and non-BEME medical education systematic reviews (Gordon et al. 2018, 2019, 2020) as a modification from its original description by Reed et al. (2005).

For the first element, studies providing sufficient data on study methodology and outcomes, we used the Cochrane risk bias tool for randomised controlled trials (Higgins et al. 2011) and the ROBINS-I tool (Risk Of Bias in Non-randomised Studies of Interventions) for non-randomised trials (Sterne et al. 2016). This is in line with the current Cochrane methodology, as described within the Cochrane handbook (Higgins et al. 2020). If no such details were provided, then the quality of the study methodology was not assessed.

For the second element, we considered whether the authors explicitly reported on five key areas related to educational development. A visual ranking system was used to report the risk of bias for these five areas: underpinning bias, resource bias, setting bias, educational bias, and content bias (Gordon and Gibbs 2014). Items were judged as being either high quality and low risk of bias (green), unclear quality and unclear risk (amber), or low quality and high risk of bias due to lack of reporting (red). This ranking system is presented in Table 1.

Table 1. Quality assessment of the interventions.

Bias source	High quality	Unclear quality	Low Quality
Underpinning bias	Clear and relevant description of theoretical models or conceptual frameworks that underpin the development.	Some limited discussion of underpinning, with minimal interpretation in the context of the study.	No mention of theoretical underpinning.
Resource bias	A clear description of the cost/time/resources needed for the development.	Some limited description of resources.	No mention of resources.
Setting bias	Clear details of the educational context and learner characteristics of the study.	Some description, but not significant as to support dissemination.	No details of learner characteristics or setting.
Educational bias	A clear description of relevant educational methods employed to support delivery.	Some educational methods mentioned but limited detail as to how these were applied.	No details of educational methods.
Content bias	Provision of detailed materials (or details of access).	Some elements of materials presented or summary information.	No educational content presented.

Thresholds for judgements were discussed before data extraction. Judgements were made within pairs, independent of each other. Disagreements were resolved through discussion or involvement of a single third author (CGC). Since no one item was considered more important than another, no weighting or overall rank was given. Instead, we presented the judgements made within each domain, so readers might assess areas of strength and weakness relating to the quality of reporting. Of importance for both elements of risk of bias was that poor reporting did not necessarily mean the educational development was of poor quality. Rather, it increased the risk that poor quality existed, hence the use of ‘risk of bias’ terminology throughout this review.

Synthesis of evidence

A descriptive synthesis of the included studies was completed using the data collected within the extraction form to summarise ‘what was done.’ Additionally, we collated the data into a range of tables and figures for easy visualisation to provide an overview of the current evidence base. A narrative of the innovations, organised into the setting of the adaptations and innovations for workplace-based clinical learning, considered the extent and range of developments, as well as the outcomes assessed.

We planned to perform meta-analysis if suitably homogenous quantitative outcome data were presented. This was not performed given the heterogeneity in outcome data across the included primary studies.

We planned to perform thematic analysis relating to how and why the adaptations in question were selected. We planned to perform open, axial, and selective coding, as outlined within our protocol (Gordon et al. 2021); however, given the paucity of published information, this was ultimately not performed.

Results

Results of the search

A total of 11,111 records were identified through database searching, with a further 23 identified through hand searching MedEdPublish. After de-duplication, 7164 records remained. After title and abstract screening, 6742 records were excluded. Four hundred and twenty-two studies underwent full-text assessment for eligibility, following which 393 were excluded, with reasons. 29 studies from this literature search were included in synthesis. We also conducted full-text eligibility assessment of the 176 studies included within the initial BEME review (Gordon et al. 2020) and the update scoping review (Daniel et al. 2021). 150 of these studies were excluded. 95 studies did not focus on remote or online learning and the 55 which did focus on these areas, were not centered on adaptations or innovations to sustain workplace-based learning. Of the 55 articles included in the final analysis within this review, one was included in the initial COVID-19 BEME rapid review (Gordon et al. 2020) and a further 25 in the update scoping review (Daniel et al. 2021). Twenty-nine studies are unique to this current review.

The flow diagram for the study inclusion process is shown in Figure 1, which includes the reasons for article exclusion at the full-text eligibility stage (PRISMA 2015). Inter-rater reliability at the screening phase for inclusion of studies was κ = 0.91, suggesting excellent alignment.

Figure 1. PRISMA flow diagram for included studies.

Supplemental Appendix 2 provides a written summary of all the primary studies included in this review. For the sake of brevity, in the results section, we have not listed specific articles if the associated data is easily identifiable in the Supplemental appendix table.

Month of publication

In the first six months of 2020, there were ten studies describing innovations or adaptations to continue workplace clinical learning (18.2%). From 01 July 2020 to 21 December 2020, a further 45 studies were published (81.8%). September was the modal month, with 13 unique publications describing interventions or adaptations as a pivot for workplace-based clinical learning. The spread of publication month by stage of medical training is charted in Figure 2(a).

Figure 2. Infographics of the key findings. (a) Month of publication. (b) Article and journal classification. (c) Origin of publication. (d) Stage of medical education. (e) Number of participants per primary study. (f) Discipline or specialty. (g) Education pivot. (h) Kirkpatrick's outcome. (i) Risk of bias assessment.

Publisher and paper length

The fifty-five included articles were published across 36 unique journals. The type of publication fell into three main categories. Seven were correspondence articles (12.7%), 15 were brief reports/innovations (27.3%), and 33 were original research articles (60.0%). Letters to the editors and brief reports/innovations were 1–2 pages in length. The original research articles were between 3 and 14 pages in length. The article and journal classification are summarised in Figure 2(b). Less than a third were published in primary education-orientated peer-reviewed medical journals (n = 17, 30.9%). The remaining 38 articles were published in specialty or discipline-specific journals (69.1%). Academic Radiology was the journal contributing the greatest number of articles (n = 8, 14.5%), followed by Medical Education (n = 5, 9.1%), and PRiMER (n = 3, 5.5%). According to impact factor (e.g. Academic Medicine, Medical Education, Teaching and Learning in Medicine, and Medical Teacher), the top-ranked medical education journals were either disproportionately represented by brief reports or not represented at all.

Classification of studies

Geographical location

Figure 2(c) presents the number of studies originating from each country. Table 2 summarises the geographical spread continentally. Forty studies were published from educators within North America (n = 40, 72.7%), with the majority from the United States (n = 38). Six studies were conducted in Europe (10.9%), five in Asia (9.1%), two in South America (3.6%), and one each in Africa and Australia/Oceania (1.8%).

Table 2. Geographical origin of included studies.

Continent	Total	UGME	PGME	Mixed
Africa	1	0	0	1
Antarctica	0	0	0	0
Asia	5	3	2	0
Australia/Oceania	1	1	0	0
Europe	6	4	2	0
North America	40	30	9	1
South America	2	0	2	0
Total	55 (100%)	38 (69.1%)	15 (27.3%)	2 (3.6%)

UGME: undergraduate medical education; PGME; postgraduate medical education; Mixed: combination of UGME and PGME

Stage of medical education

Figure 2(d) shows the number of studies according to the stage of education. The majority of innovations and adaptations were deployed within undergraduate medical education (n = 38; 69.1%). Postgraduate learners were the focus within 15 studies (27.3%). Two studies involved a mix of undergraduate and postgraduate learners (3.6%).

Number of participants

Figure 2(e) graphically presents the range of included learners amongst the 48 studies which presented a precise number of participants. The mean number of participants was 53.7 and the range was 2 (Bautista et al. 2020; Shibata et al. 2020) to 610 (Trujillo et al. 2020). Seven studies did not provide the number of participants.

Discipline or specialty

Table 3 and Figure 2(f) show the disciplines and specialties represented by the studies, sub-categorised by the education stage. Radiology was the most frequently described specialty across both undergraduate and postgraduate learners, with eight and four studies, respectively. Paediatrics was the second most frequently described specialty, with three studies within UGME and two studies within postgraduate medical education. Internal medicine and primary care were covered by four studies each, all of which were based in UGME. There was a paucity of literature involving medical specialties within the postgraduate setting, with just two articles describing innovations or adaptations. This was in contrast to 13 articles involving undergraduate learners, spanning eight medical specialties. Two studies did not provide the specialty or discipline where the educational innovation or adaptation was deployed.

Table 3. Discipline or specialty of deployed educational innovation or adaptation.

	Total	UGME	PGME	Mixed
Surgical
Cardiothoracic surgery	1	1	0	0
General surgery	2	1	1	0
Neurosurgery	4	1	2	1
Obstetrics and gynaecology	1	1	0	0
Otolaryngology	2	0	1	1
Urology	2	2	0	0
Vascular surgery	1	1	0	0
Medical
Cardiology	1	0	1	0
Dermatology	2	1	1	0
Emergency medicine	3	3	0	0
Infectious diseases	1	1	0	0
Internal medicine	4	4	0	0
Oncology	2	2	0	0
Neurology	1	1	0	0
Palliative care	1	1	0	0
Other
Inter-professional	1	1	0	0
Pathology	3	2	1	0
Radiology	12	8	4	0
Primary care	4	4	0	0
Paediatrics	5	3	2	0
Not stated	2	0	2	0
Total	55 (100%)	38 (69.1%)	15 (27.3%)	2 (3.6%)

Summary of educational developments

All the included studies involved an adaptation or innovation to maintain learning that was previously delivered within the clinical workplace. Table 4 and Figure 2(g) charts the eight categories of focus for the educational adaptations and innovations. A small number of studies were classified into more than one section. The focus of the developments, in descending order by representation, included:

Table 4. Focus of education adaptation or innovation.

	Total	UGME	PGME	Mixed
Online learning	33	26	7	0
Remote clinical interaction as an adaptation in response to a change in practice	9	7	2	0
Online simulation	8	3	5	0
Remote clinical interaction as an innovation to support COVID-19 service provision	6	6	0	0
Remote adaptation of multidisciplinary ward round	6	3	2	1
Remote adaptation of multidisciplinary team meetings	2	1	1	0
Live-streaming surgery or procedures	1	1	0	1
Remote clinical interaction as a replacement	1	1	0	0

• Adaptation to online learning (n = 33, 61.8%).

• Online simulation (n = 9, 16.4%).

• Remote clinical interactions as an adaptation in response to a change in healthcare practice (n = 8, 14.5%).

• Remote clinical interactions as an innovation to support COVID-19 service provision (n = 6, 10.9%).

• Remote adaptation of multidisciplinary ward rounds (n = 6, 10.9%).

• Remote adaptation of multidisciplinary team meetings (n = 2, 3.6%).

• Live streaming of surgery or procedures (n = 1, 1.8%).

• Remote clinical interaction as a replacement (n = 1, 1.8%).

There was a disparity in the focus of the educational adaptations based on whether the learner was an undergraduate or postgraduate. The majority of undergraduate pivots were situated within online learning (n = 26, 68.4%). A much smaller proportion of studies within undergraduate settings described the adaptation to remote clinical interactions, either as a response to a change in healthcare practice (n = 7, 18.4%) or to support COVID-19 service provision (n = 6, 15.8%).

A different pattern across PGME was seen. Online learning was utilised in nearly half of all postgraduate deployments (n = 7, 46.7%) compared to 68.4% of studies within UGME. Online simulation was also a more frequently utilised modality in postgraduate learners in comparison to undergraduate students (n = 5, 33.3% versus n = 3, 7.9%). All six deployments involving remote clinical interactions to support COVID-19 service provision were within UGME.

Online learning

Thirty-three studies involved an element of online learning (60.0%). Twenty studies presented an educational offering which combined asynchronous and synchronous learning opportunities (60.6%). Eleven studies described exclusively synchronous learning (33.3%). Two studies described a purely asynchronous learning opportunity (6.1%). Twenty-three of the 33 studies involving online learning did not involve any form of remote contact with patients or the clinical workplace (69.7%), making online learning the only educational component in the majority of studies.

Synchronous learning opportunities

Thirty-two studies described a synchronous component to their educational intervention. Eighteen studies involved real-time lectures, 11 included interactive tutorials, six required students to deliver journal club presentations, four incorporated quizzes, and three involved videos. Synchronous individual or small-group case reviews via teleconferencing software were presented in 13 studies.

Asynchronous learning opportunities

Twenty-two studies presented an asynchronous component to their education intervention. Six studies involved pre-recorded lectures, seven studies involved videos produced from other organisations/educators, 13 studies involved reading and other widely available online learning resources, and two studies used online discussion boards.

Seven studies utilised pre-existing learning modules, which were widely available prior to the COVID-19 pandemic. These included the American College of Physicians modules on telemedicine (Abraham et al. 2020), Health Education England (NHS) modules on dermatology (Ali et al. 2021), and Centre to Advance Palliative Care training modules on end-of-life care (Kahn et al. 2021).

A wide range of learning methods were adopted across studies involving asynchronous educational activities. Within these, there were limited evaluations of the comparative utility of such methods. In Coffey et al. (2020), students reported that the use of question banks for learning, amongst a range of learning resources, were the most valuable. The authors felt the reason for this was increased interactivity within active learning modalities, such as question banks and case-based discussions, in which faculty were able to give feedback to students, identify and address deficits in knowledge, and help students avoid digital fatigue.

Virtual case discussions

Radiology images or pathology slides from real patients were used within eight studies. One study described the remote transition of the radiology ‘read-out’ (Matalon et al. 2020), which involved the reporting of live radiological investigations from patients admitted to the hospital. The radiology readout was described as similar to a ‘grand round’ discussion and is considered to be equivalent to ‘rounding on patients’ or ‘case-based discussion’ and is a fundamental academic practice involving the radiology trainee and trainer where active learning occurs, personal connections are made, and work is accomplished (Dako and Awan 2021).

Three studies in radiology and one in radiation oncology utilised archived radiological images from the host institution to continue clinical learning. Two of these radiology studies involved synchronous approaches to virtual case discussions for residents but involved archived images rather than examinations from current in-patients (Recht et al. 2020; McRoy et al. 2020). Within the latter, the aim was to ‘mimic’ a typical working day, with an expectant proportion of normal and abnormal cases, with the recognition that daily read-outs were a vital component of resident radiology training. In Kahn et al. (2021), radiation oncology elective students performed synchronous image contouring and case reviews using archived images under faculty guidance.

Two studies described the use of digitised pathology slides within asynchronous learning activities for undergraduate reporting. In Samueli et al. (2020), medical students recorded diagnoses asynchronously within an online quiz via Moodle. A second study described the use of remote pathology slide access to support a shift from an in-person pathology residency to a hybrid on-site and remote training model (Chin et al. 2021). In this study, respondents indicated that the remote experiences were less effective for reviewing cases and generating reports and was felt by the authors to relate to the lack of ‘live cases.’

McRoy et al. (2020) described the use of Google Drive for postgraduate residents to submit independent interpretations for clinical cases. The authors recognised the comment function within Google Docs to be a valuable tool in fostering constructive and collaborative discussion amongst facilitators. An important limitation recognised was the difference between active workstation learning and reviewing select static images, including search patterns, windowing, and quality control. Within another radiology elective (Alpert et al. 2021), where faculty preceptors curated lists of examinations, high-yield cases were selected, and were encouraged to avoid ‘zebras’ – unusual diagnoses that the students may never see again. In the same study, medical students reported an ‘equal or greater sense’ of student involvement and educational value compared to the conventional in-person reading room experience, despite the number of contact hours dropping from 20 (conventional) to 3 (virtual).

Videos for education

Videos were used as a component of the deployed education within nine studies. Four studies described the use or production of videos from within the host institution. Yang et al. (2021) utilised pre-recorded video instruction prior to a paediatric simulation session for undergraduate medical students. In three studies, pre-recorded videos were utilised within synchronous learning activities. Dong et al. (2021) described the use of pre-recorded surgical procedures within an undergraduate cardiothoracic surgery elective where faculty ‘walked students through’ surgical cases to a maximum group size of five students to facilitate ‘active conversation’ and ‘case-based problem solving.’ In the UK, Dow et al. (2020) described the use of recorded real-life general practitioner (GP) consultations followed by GP facilitated case-based discussion. In DePietro et al. (2021), medical students observed recorded resident-produced presentations of cases from the previous day.

Interaction within online learning developments

Four papers commented on the lack of student-patient interaction as a barrier to their learning opportunities. Coffey et al. (2020) felt it was difficult to adapt clinical education online due to a lack of guidance, reporting fear in students of lacking practical knowledge and skills. Whilst they thought online strategies were helpful, they stated that clinical experiences remain essential. Kahn et al. (2021) agreed, stating their elective may augment face-to-face training but lacked impact due to a lack of in-person interaction with cancer patients and their families. Shin et al. (2020) described the lack of operating room experience as a ‘major drawback’ of their surgical elective. Yang et al. (2021) also made reference to including more exposure to actual patients in future electives.

Nine papers reported a decrease in student-faculty interactions as a barrier to their innovation. Adams et al. (2021) and Alpert et al. (2021) reported that students were unable to gain an understanding of what a ‘day in the life’ of a radiologist is like. DePietro et al. (2021) suggested that this lack of interaction could impact the students’ ability to obtain letters of reference for future applications. Krawiec and Myers (2020) reflected that they did not feel students received enough individual attention during their innovation. Alpert et al. (2021) argued that the loss of student interactions with graduate trainees is a limitation to remote learning, given the great value placed in fellows and residents as an educational resource. To combat this, Dong et al. (2021) ensured faculty were consistent for small group sessions. Nagji et al. (2020) utilised social media platforms with the aim of placing faculty and students on a ‘level playing field,’ which reduced feelings of isolation for learners. They also hosted a live ‘ask me anything’ webinar between students and faculty.

Dedicated time for ‘unplanned’ conversation was mentioned in three papers: Chin et al. (2021) included a regular coffee break to allow for free-flowing conversation between students and faculty; Gomez et al. (2020) included regular ‘office hours’ to enable students to ask questions freely or seek advice, and Durfee et al. (2020) created ‘homerooms’ for students which met regularly and were hosted by the same faculty to allow for continuity and mentorship. Of note, Gomez et al. (2020) found they had both increased numbers of students and higher interaction following their online adaptation. They also noted that radiology was an ideal field for remote learning.

Telemedicine curriculum

Two papers focused on the abrupt increased requirement for telemedicine during the pandemic; on a background of gradual growth in the telemedicine industry over the past two decades. They reported a lack of formal, standardised curricula in this area. Abraham et al. (2020) utilised a widely available e-module from the American College of Physicians and combined this with student involvement in telehealth clinics. Ha et al. (2020) developed a telemedicine curriculum. Both recommended the development and incorporation of a standardised telemedicine curriculum within undergraduate and postgraduate training.

Remote clinical interaction as an adaptation in response to changes in clinical practice

Eight studies described innovations or adaptations accompanying a change in clinical practice, typically relating to the transition of clinical consultations into the online space. Six studies were deployed within UGME, universally for clinical medical students. Three studies involved remote clinical interaction as the sole educational activity within the deployment. The remainder integrated clinical interactions amongst synchronous and asynchronous online learning. Seven of the eight studies involved, in some form, a teleconsultation between learner and patient.

All the telehealth consultations described across seven studies were delivered using a range of communication tools, including Zoom (Zoom Video Communications Inc.), telephone, AccurRx (AccuRx Ltd), Epic system (Epic Systems Corporation) and Teams (Microsoft Corporation). Across the included studies, there was a paucity in justification around the specific choice of telecommunication platforms used. Abraham et al. (2020) documented the need for a mobile number masking application when conducting telehealth consultations over the phone.

Clinical interaction summaries

Of the seven studies involving teleconsultations, all except one involved interaction between learner and patient. In Lal et al. (2020), students observed 12 half-days of outpatient telepalliative care clinics, where they learned through observation rather than interaction. In the other six studies, students actively participated in the telehealth consultation. Abraham et al. (2020) described students attending a once-weekly telehealth clinic, where they remotely ‘shadowed’ the history and physical performed by the attending physician and performed a scripted social determinant of health screening. Michener et al. (2020) described as part of a multimodal virtual geriatrics elective, a telephone interview with an older adult patient or relative. Huffman et al. (2020) described postgraduate paediatric fellow participation in telehealth consultations, where the fellow led the consultation, with faculty observation. All studies apart from one involved direct supervision by faculty members, who either participated in, or observed, the remote clinical interactions. In the study described by Michener et al. (2020), participants contacted elderly patients via telephone, independent of faculty.

Benefits and challenges

Despite being successful in achieving the overall goal of continuing clinical workplace learning through teleconsultations and remote reporting of active cares, several studies documented challenges associated with these approaches. Technical issues were a minor issue and only reported by one study (Cain et al. 2020). This was likely related to established telehealth practices and growing familiarity and experience with such tools. One telehealth study noted limitations of pre-existing software, specifically multiparty functionality (Huffman et al. 2020). Darnton et al. (2021) cited student fear of being disturbed or overheard by third parties as a ‘weight of responsibility,’ although this highlights the extensive lengths students described to protect confidentiality. In the same study, the three-way consultation between student, doctor (faculty), and patient, was felt by faculty to alter the previously clear lines of communication. For the studies involving telephone consultations, students commented on the lack of real-time observation as a particular downside.

Darnton et al. (2021) commented that feedback providers were less likely to be distracted when remotely supervising a video consultation, as it required them to concentrate throughout. The same students also felt the inability to perform a physical examination necessitated higher quality history-taking skills to obtain the diagnosis, making teleconsultation a valuable tool in developing communication skills. Huffman et al. (2020) described shared beliefs of fellows and faculty, stating that feedback had been enriched and that new opportunities had emerged for confidential feedback during the visit (using the chat functionality) and after the visit (using planned debrief). This was a sentiment echoed by the learners within Abraham et al. (2020), who expressed appreciation of timely feedback during telehealth encounters, where the resident or attending physician ‘stayed on the line,’ to provide immediate feedback. In each study involving teleconsultation as part of the deployed education, the teleconsultation was consistently recognised as the most valuable component. Sandhu et al. (2020) recognised that delivery of a virtual clerkship, which involved teleconsultation, would allow visiting students to attend.

Remote or modified clinical interaction as an innovation to support COVID-19 service provision

Six studies described interventions in which learners participated in activities to support clinical service. All papers in this category involved undergraduate medical students. No study explicitly stated that the medical students were paid for these activities. Half of the deployments were introduced as components of the curriculum (n = 3). The remaining three studies were conducted with the medical students learning within a voluntary capacity.

Remote interaction to support service provision

The majority of services students contributed to were designed to support patients with COVID-19. Others provided additional patient support such as screening for domestic violence and performing medication reconciliation (Bautista et al. 2020); preparation for dermatology clinics (Belzer et al. 2021); follow-up calls for patients seen in the Emergency Department to address COVID-19 or general medical patient complaints using a script and checklist (Chandra et al. 2020); and delivering food parcels to families (Daccache et al. 2020).

Supervision

Student-patient contact was supervised in several ways. Two studies described a discreet student-led service. The remaining four studies described some form of remote supervision. One study described faculty availability to discuss cases if the learner referred a patient for medical assessment (Carson et al. 2020), but the learners were mainly supported during the calls through a continuously updated ‘script.’ Server et al. (2020) described ‘only indirect supervision,’ recognising this as a limitation of the voluntary service. Bautista et al. (2020) described that students conducted telephone calls independently, with preceptors reachable over personal mobile phone for questions offline or to join the call if required. The outreach Zoom consultations were followed by debriefing sessions to ‘work through clinical reasoning’ with preceptor supervisors. In one study, students were encouraged to assist with the documentation of callbacks (Chandra et al. 2020).

Remote clinical interaction as an adaptation

This category refers to any remotely delivered clinical education that had previously taken place face-to-face before the COVID-19 pandemic. One study described the use of high-definition video cameras and Zoom teleconferencing software to facilitate live interactions between patients and medical students learning within a neurological clerkship (Tsang et al. 2020). This study was unique in that it utilised inpatient encounters, where students virtually directed a tutor at the bedside in conducting a neurological examination. This intervention was well received by learners and deemed comparable to traditional bedside teaching, except in fulfilling physical examination skills outcomes. This limited remotely conducted examination using a proxy (the tutor) provided useful discussion points, and learners were introduced to the principles of telehealth etiquette.

Remote adaptation of a multidisciplinary team meeting

This category refers to learning opportunities afforded by the pivot from face-to-face to remotely conducted multidisciplinary team meetings. Two studies fell into this category. Bandi et al. (2020) described the participation of otolaryngology residents in a weekly virtual multidisciplinary tumour board, delivered with the same frequency as before the COVID-19 pandemic. The specific role of residents in the tumour boards prior to the virtual adaptation is not delineated. Within the described intervention, residents prepared clinical cases for discussion, distributed the list to participants for information, and participated in the board discussion and “learnt from it”. An attending physician provided supervision, but no further detail around the nature of this supervision was provided. Sandhu et al. (2020) described the opportunity for medical students on a virtual radiation oncology clerkship to attend multidisciplinary tumor boards/meetings and chart rounds but did not describe the specifics of what participation entailed. Attendance at the multi-disciplinary team (MDT) meeting, which was held via WebEx (Cisco Webex), was a supplementary activity to synchronous didactic lectures, virtual clinics, and virtual journal clubs.

Remote adaptation of multidisciplinary ward round

This category refers to the educational opportunities afforded by remotely conducted multidisciplinary ward rounds which were previously face-to-face prior to the COVID-19 pandemic. Six studies described techniques to facilitate remote undergraduate and postgraduate learner inclusion in multidisciplinary ward rounds.

Technical requirements

All studies utilised live-streaming of video and audio feeds from bedside clinical interactions via video-conferencing platforms. Three studies demonstrated the use of mobile workstations equipped with cameras and microphones and live-streamed the media feed via video-conferencing platforms. Custer et al. (2021) adapted standard bedside rounds where members of the MDT participated remotely from a conference room, which included critical care residents. Here the mobile workstation was equipped with a high-resolution camera, targeted microphone and speaker system, a monitor and a basic computer. As described by Hofmann et al. (2020), the mobile workstation was less costly, where an iPad Pro was fixed to a computer on wheels using an elastic exercise band, following which the live media was streamed via Zoom to medical clerkship students. The setup described by Salcedo et al. (2020) consisted of a cart equipped with devices to allow the streaming of video, audio, and clinical data, such as auscultation sounds, vital signs, fundoscopy, otoscopy, bedside ultrasound, and macrovideo of physical signs.

Pennell et al. (2020) described the use of a mobile phone to facilitate live streaming to medical students, where the mobile device was directed towards the patient and clinician during the consultation. Munusamy et al. (2021) utilised a pair of smart glasses, costing $1800 apiece, which permitted a remote interaction between a socially distanced specialist and a colleague wearing the smart device at the patient bedside.

Student–faculty–patient interactions

Five studies permitted remote interactions between socially distanced learners, patients, and facilitators at the patient bedside. One study only permitted remote interaction between a socially distanced specialist and the colleague donning the smart glasses at the patient bedside but did not permit for interaction between remote learner and the patient (Munusamy et al. 2021).

Benefits and challenges

Learners valued the opportunity to maintain clinical encounters and clinical learning that was otherwise precluded by social distancing measures. The ability to maintain clinical supervision through remote interaction was equally noted (Munusamy et al. 2021). Limitations included a lack of interaction with the wider multidisciplinary team and family members (Custer et al. 2021); bandwidth and WiFi limitations such as latency (Hofmann et al. 2020); the need for training with novel technology (Munusamy et al. 2021); prolonged duration of the ward round, the inability for learners to conduct physical examinations, potential inability for learners to engage in the informal curriculum including professional values and behaviors (Pennell et al. 2020); patient discomfort with live-streaming to remote learners (Salcedo et al. 2020); and maintaining patient confidentiality (Custer et al. 2021). Several studies referred to the importance of patient confidentiality and the value of a daily teleconferencing etiquette brief.

Online simulation

Nine studies involved simulation delivered remotely as an adaptation or innovation to replace workplace-based clinical learning. There was a diversity of approaches, not only in how simulation was delivered, but also how it was integrated amongst other educational activities. Across the board, simulation involved the use of virtual reality patients augmented with artificial intelligence, remote laboratories for the development of surgical and procedural capabilities, mobile-based simulation platforms, and live-streaming from a simulated patient environment.

Virtual reality

One study described the use of virtual reality innovations. In De Ponti et al. (2020), as a described replacement for the bedside apprenticeship model of education, the Body Interact platform (Body Interact, Take the Wind, S.A) was used to simulate clinical scenarios for final year medical students. Within each two hour session, students spent 40-minutes accessing the platform online and worked independently on the clinical scenarios before returning to Microsoft Teams for a debriefing session to critically discuss the virtual case with peers and a tutor. Whilst the learners found the simulation experience realistic, useful and meeting expectations, no participant recommended the standalone use of virtual patient simulation when given the choice of simulation only, traditional training only, or a combination of both.

Remote laboratories

Five studies employed the use of remote skill laboratories. Gallardo et al. (2020), described a home training skills programme for neurosurgical residents, which set out to improve microsurgical skills using a smartphone camera and low-cost materials. The authors comprehensively reported the exercises involved, including full resource details, and cited an urgent need to create alternative solutions to ensure trainee development, with a particular consideration around cost and lack of personally owned microscopes. To evaluate learning, participants submitted data and videos at baseline and post-course, which demonstrated significant improvements with both dominant and non-dominant hands across all areas of training.

Pasricha et al. (2020) described a prospective randomised study, where residents were placed into remote feedback or no remote feedback groups. The authors found that for the educational task of corneal suturing, the use of a remote wet lab was as effective as previous in-person wet labs. Shibata et al. (2020) described an ‘Ultrasound Treasure Hunt,’ where novice learners performed ultrasonography on themselves and the people they were in direct with, against step-by-step instructions and a ‘treasure map.’ Image files were then uploaded for faculty to view, where feedback was provided using video conferencing software with screen sharing functionality. The authors recognised the limitation of this educational model in that there was a lack of real-time feedback during image acquisition and a lack of significant pathology during self-imaging. In Singh et al. (2020), simulation-based echocardiography tasks were performed using the 3 D systems ultrasound mentor simulation program and mannequin and involved the use of a preset checklist of basic competencies.

Mobile-based simulation platform

Jarry Trujillo et al. (2020) described the adaptation of a web-mobile-based platform for procedural skill learning, including paracentesis, thoracentesis, intubation, and endoscopy. Within this model of education, trainees watched instructional clips on their phone, practiced at their own pace and then uploaded a video of themselves performing the procedure. Following upload, and within 72 h, learners received expert evaluation and feedback via the application. This innovation was recognised by the authors as removing the need for on-site trainers, provided convenience to both instructor and trainee, and was easily scalable.

Live-streaming simulation

Yang et al. (2021) described the use of existing simulation centre facilities (faculty, staff, and resources) to deliver simulated sessions via Zoom to medical students learning within a paediatrics elective. As an important outcome of this deployment, the authors intended to apply telesimulation as a method to augment standard in-person medical student simulation activities after the pandemic.

Live-streaming surgery or procedures

This category refers to the use of live video and audio streaming to facilitate remote learner observation of surgeries or procedures that were previously attended in person prior to the COVID-19 pandemic. One study, reported by Jack et al. (2021), described the use of multiple video capture devices, including a GoPro camera with a headband adapter, a camera integrated into the overhead surgical lights, and a camera mounted to a microscope, to stream the operating room live via Zoom to neurosurgical residents and medical students. Audio communication between the primary surgeon and learners was established with a wireless Bluetooth headset. This study provided full details of the hardware, software, and technical considerations required to deliver live-streaming learning events. Participants valued the picture and audio quality and noted improved knowledge of neuroanatomy as a result. The authors commented on the limitations imposed by the zoom capability of different cameras and the subsequent limited visualisation of the surgical fields. Furthermore, the dependence on internet speeds and potential safety and infection control hazards associated with the required cabling were also recognised as a limitation.

Kirkpatrick outcome

85.5% (n = 47) of included studies documented Kirkpatrick level 1 outcomes (satisfaction or reaction). 27.3% (n = 15) of included studies documented Kirkpatrick level 2a (change in attitudes) and 30.9% of studies documented Kirkpatrick level 2b (changes in knowledge or skill). No study reported outcomes at Kirkpatrick level 3 or 4. Kirkpatrick outcomes are summarised in Figure 2(h).

Quality assessment/risk of bias

There was one randomised-controlled trial included within this systematic review (Pasricha et al. 2020) (Table 5). There was no clear detail about the randomisation processes or blinding for outcome assessment, thus ‘unclear risk’ of bias. Since this was an open-label trial, the risk of performance bias was considered ‘high risk.’ Additionally, the inclusion of just ten non-matched participants further increased the likelihood of bias. There was a ‘low risk’ of bias for outcome data given the completeness of data presented. All other studies within this review were action-based studies, rather than study methodologies that incorporated elements of randomisation or assessed outcomes comparatively between an ‘intervention’ and ‘control’ group. No studies fell into the case report category. As such, ROBINS-I evaluation was not undertaken to assess the methodological quality of these studies.

Table 5. Risk of bias assessment for randomised-controlled trials.

Risk of bias	Pasricha 2020
Random sequence
Allocation concealment
Blinding of participants
Blinding of outcome assessment
Incomplete outcome data
Selective reporting
Other bias

Green – low risk of bias; amber – unclear risk of bias; red – high risk of bias

Quality of reporting for the educational developments

The risk of bias framework for the reporting quality of the developments was applied to all articles and is summarised in Figure 2(i). There was no study considered at low risk of bias in all five domains.

Of the 55 studies, only four studies provided a clear and relevant description of the theoretical models and/or conceptual frameworks underpinning the development (7.3%). The vast majority of studies (n = 45, 81.8%) failed to refer to any theoretical models. Seventeen studies (30.9%) provided a clear description of the resources required to deliver the innovation or adaptation. All studies, bar one, provided at least some description of the educational context and learner characteristics (98.2%). Thirty studies provided a clear description of the educational context, making this domain the highest quality across the five indicators of reporting quality (54.5%). Three studies (5.5%) provided a clear description of the relevant educational methods employed to support delivery. Seventeen studies made no reference to relevant educational methods (30.9%). Fifteen studies provided full details of materials, as to encourage replication (27.3%). Seven studies did not present any educational content (12.7%). Across all five domains, theory and educational methods were the least well reported.

Two articles stood out in terms of reporting quality, both with four ‘low risk of bias’ assessments and one ‘unclear risk of bias’ assessment (Nagji et al. 2020; Samueli et al. 2020), signifying significant clarity in terms of reporting in these areas.

Discussion

There is a rapidly expanding literature in all areas of COVID educational impact, with 55 included papers in this review about adaptations and innovations to supplement, sustain or replace clinical workplace-based learning. Despite the enormity of the challenges and the rapidity through which adaptations and innovations in clinical medication were deployed, the creativity and adaptability of educators were evident throughout this review, reflecting the ever strong will to continue the provision and development of excellent medical education.

The majority of publications were rapidly disseminated reports, which is likely to reflect the context in which these reports were produced. Most were published within clinical journals, reported initial learner satisfaction or evaluation, with 22 articles (40%) being correspondence articles or brief reports (1–2 pages in length). We found it counter-intuitive that the studies reported as brief reports or letters to the editors often described the most innovative and potentially practice-changing innovations, however this may not reflect deficiencies in the traditional articles being published, but instead the time lag for more detailed studies, as they require a more extensive and time-intensive peer review.

It was notable that adaptations to support continued clinical learning for undergraduate medical students were more than twice as common as those for postgraduate trainees. The majority of innovations for undergraduate learners involved moving clinical learning to online settings using, typically, a combination of synchronous and asynchronous activities. The relative lack of papers describing postgraduate innovations may be due to the dual roles of postgraduates both as learners and as service providers. Consequently, they may have been more likely to have continued to contribute to face-to-face care, particularly within inpatient settings, with sustainable opportunities for workplace-based learning.

Most reported learning was developed in response to local contexts, needs, constraints and resources. The descriptive synthesis within the results offers a comprehensive insight into the range of choices made, some of the reasons why such decisions were made, and hopefully, some of the potential or actual problems that had to be overcome. Whilst some of these may be considered minor barriers, awareness and consideration of these by clinical teachers a priori offers the best chance to deliver excellent education to learners. Examples include the implementation of planned virtual or online ‘coffee breaks’ to allow for free-flowing and highly valuable unstructured discussions that often occur during face-to-face interactions and the growing awareness that students who are using electronic devices to participate in patient interactions must have access to software to anonymize their own personal details.

The absence of a standardised curriculum for telemedicine consultations was noted and given the role teleconsultation will have in the provision of healthcare moving forward, it warrants further exploration.

The advent of live-streaming technologies, progression of simulation into virtual formats, and remote consultation are prime examples of how technology enabled continued workplace-based learning. The most valuable tool available across the described innovations and adaptations was the use of teleconsultation platforms to sustain clinical learning, which was convenient, accessible, and aligned with change in healthcare practice. A key theme across the studies describing remote consultations was enhanced opportunities for students to receive feedback. The use of the chatbox function, online checklists of performance, and remote debrief were recognised as valuable in enabling students to receive real-time feedback from focused clinical supervisors. The use of online learning strategies, particularly the use of teleconsultation, permitted a more standardised and equitable educational experience across learners, whereby students received the same educational materials, equivalent clinical experiences, and equal access to opportunities that may have previously been first-come, first-serve, or dependent on student initiative. With this considered, we note a lack of published work from low-resource settings, and the degree to which the described interventions are readily translatable to such settings remains undetermined.

Quality of study methodology and reporting

In the year since the COVID-19 pandemic began to impact medical education, the overall quality of the evidence base in terms of both study methodology and reporting has only modestly changed (Gordon et al. 2020; Daniel et al. 2021). The included studies are limited in any discussion of theory or frameworks. In normal circumstances, this would be key in understanding ‘why’ such an adaptation was made and as such, guide dissemination. However, there is a further clarification question related to this that is key within this context. Educators need to know ‘how’ to deploy the complex interplay of pedagogical options to achieve such pivoted online learning. The range of innovations presented within the work are so broad that details of such are vital to facilitate the complex design considerations needed and to inform how researchers overcome these unique challenges and contextual factors. This can include the choices of software design, decisions of synchronous vs asynchronous elements of delivery, simulated vs authentic materials, group dynamics before, during, and after such educational encounters, debrief and reflection methods. The options are infinitely complex. This is a considerable gap in the literature, with most studies presenting methods as if they were dichotomous, representing only the parameters for a pivot vs the conventional approach.

The issues with risk of bias in reporting of the adaptations themselves is a far greater issue of concern. It is still unclear whether this risk of bias represents just that – a risk related to poor reporting, or whether this is related to actual limitations in adaptation and innovation design and deployment. However, this still leaves educators and researchers in the difficult position of having a rapidly growing evidence base that has a significant paucity of evidence to guide future replication and dissemination of such works. This is an issue of urgent importance and authors of the primary literature, as well as editorial boards in journals, must strive to ensure this is addressed moving forward.

It could be assumed that when workplace-based clinical learning interactions represented a direct adaptation of an identical clinical learning interaction from face-to-face to online, that the primary pedagogical principles do not change. This review demonstrates that this is a glib interpretation. Instead, there are clearly levels of complexity and interaction here that move past how online learning has been most commonly conceptualised in the field (O’Doherty et al. 2018). Rather, as this review describes, the need to access authentic clinical learning interaction – the pedagogical principles have to be layered. The methods of teaching deployed and any revisions or shifts in approach due to the new online approach are key. The specific method of remote access itself introduces another level of complexity. However, it is the unique and symbiotic interplay of both of these factors that is unique within each of the included studies and arguably of most interest to clinical teachers and researchers. In many ways, this can be considered one of the easiest elements to be included in published studies, but a considerable gap remains.

Strengths and limitations

Despite the rapid timeframe through which this review was conducted, the underlying principles of systematicity and methodological rigor were not compromised and thus should be recognised as a strength of this review. The authoring group represented an international collaboration including key stakeholders, such as medical students, faculty educators, and practicing clinicians from a range of specialties, including paediatrics, histopathology, primary care, and emergency medicine.

This review has several limitations. We limited our search to four databases to align with previous reviews, which opens up the potential for missed developments. Whilst we performed a manual electronic search of MedEdPublish, we did not hand search all non-indexed medical education journals. Nor did we search the reference lists of included studies for additional potentially relevant citations. Given the rapidity through which the evidence-base is evolving, by the time this review is published, it is likely several additional articles will have been published, possibly addressing the range of study methodology and reporting quality elements discussed throughout. We also recognise, given the focus placed on medical education involving undergraduate medical students and graduate doctors, that the findings of this review may not be directly applicable or transferable to research or practice within other health disciplines. A further limitation relates to our reporting of the education interventions, context, and research methods employed. We recognise that reporting on a substantial body of research describing complex educational interventions within the constraints of a BEME review has potentially limited the expansiveness of our reporting and discussion. We have provided extensive data within our results tables and appendices in an attempt to mitigate against this and hope the detail contained within these resources offers a comprehensive render of the research included within this review.

Implications for practice

This review has highlighted the extensive continued development of online, remote educational designs to facilitate undergraduate and postgraduate clinical learning. Whilst the relative advantages and disadvantages of the adaptation to online learning for elements of the formal curriculum have been demonstrated, there is a paucity of work exploring the manners in which the informal and hidden curricula may be represented online. Future educational interventions could consider the manners in which important educational phenomena such as mimetic learning, role modelling, professional identity development and reflective practice can be replicated in remote learning environments. Similarly, novel considerations to promote remote interactions with the wider, interprofessional clinical team, as well as the development of remote interpersonal skills with patients and their families, could be expanded upon in future work. Although the technological requirements and applications to facilitate remote learner interaction in live-streamed ward rounds and clinics have been described, further work to maximise the educational utility of such remote interactions would aid the dissemination of these methodologies. Mechanisms which utilise low-cost, widely accessible technology and permit equitable access to asynchronous and synchronous learning resources will be particularly valuable.

More broadly, the sustainability and long-term outcomes for learners and patients are yet to be addressed in relation to remote educational designs. We are at a critical point to determine how evidence and scholarly pedagogy inform future plans. Possibilities exist to revert to ‘pre-COVID’ models, develop hybrids, or develop a remote focus on some areas of teaching. This is of particular importance given the increasing numbers of medical students and demands placed upon access to high-quality clinical experiences. Many of the innovations and adaptations described in this review may be suited to further maximising the educational utility of limited clinical interactions.

Implications for research

The issues of clarity in research method reporting and intervention reporting have both been clearly demonstrated. They must be addressed by future works in the field to ensure the iterative evolution of the evidence base as a clear tapestry of choices for educators, rather than as distinct. There are examples highlighted in this review of theoretically informed, high-quality research. Most, however, are single-centre evaluations examining learner satisfaction.

Future research studies must consider ‘how’ and ‘why’ adaptations and innovations to remote educational strategies are useful, given the heterogeneous and diverse approach to pivoting clinical workplace learning. This can be achieved by clearly describing the educational context, including the pre-existing learning opportunities and reasons for innovation or adaptation, the methods employed to select the pedagogical approach, full description of resources (both educational and operational), and the considerations made through educational methodology and delivery.

Given the reality that large cohorts of medical students have missed out on varying degrees of authentic clinical environments, the associated pressures, and nuanced learning, there is the potential these learners may be left underprepared for independent practice upon qualification (Dornan et al. 2015). Within the second and third waves of the pandemic, many educational organisations returned their learners to clinical workplaces in order to overcome this loss of experience. In due course, we anticipate there will be a collection of articles describing innovative approaches to continued workplace-based learning within the contexts of social distancing and risk mitigation to learners and patients. Specifically, we anticipate more international primary studies originating from countries outside of the USA, which was notably low in number within this review.

Conclusions

Since the beginning of the COVID-19 pandemic, educators across the globe have instigated new methods of education delivery to continue workplace-based clinical learning. As we move into the ‘new normal’ within medical education, it is important to utilise our collective learning and apply this within future educational deployments, regardless of whether medical education reverts to pre-pandemic practices, or involves a combination of traditional practices and innovations. We would encourage medical educators, authors and peer-reviewers, to incorporate explicit reporting of all outcomes, including the relevant educational theory into publications and peer review processes. In doing so, it will allow us to gain an understanding of ‘which’ elements of educational adaptations and innovations have utility and ‘how’ and ‘why’ they work, which is essential in progressing the evidence-base forward.

Acknowledgements

The authors would like to thank Whitney Townsend for her expert input into the search strategy and assistance running the database search.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Additional information

Notes on contributors

Ciaran Grafton-Clarke

Ciaran Grafton-Clarke, MBChB, is an Academic Clinical Fellow in Cardiology at the University of East Anglia, Norwich, UK.

Hussein Uraiby

Hussein Uraiby, MBChB, PGCert, FHEA is a Specialty Trainee in Histopathology and a Clinical Education Fellow at the University Hospitals of Leicester NHS Trust, Leicester, UK.

Morris Gordon

Morris Gordon, MBChB, PHD, MMed, is Cochrane Coordinating Editor, Chair of the BEME Executive Committee, and a Professor of Evidence Synthesis and Systematic Review, University of Central Lancashire, Preston, UK.

Nicola Clarke

Nicola Clarke, MBChB, PGCert, FHEA is an Academic Clinical Fellow in General Practice and an Honorary Lecturer at University College London Medical School, London, UK.

Eliot Rees

Eliot Rees, MBChB, MA, is a Lecturer in Medical Education at Keele University, an NIHR Academic Clinical Fellow in General Practice at University College London, and Chair of Trainees in the Association for the Study of Medical Education (TASME), London, UK.

Sophie Park

Sophie Park, MBChB, M.Med.Sci (dist), EdD FRCGP, SFHEA, is a General Practitioner and Professor of Primary Care and Medical Education. She is an NIHR School of Primary Care Evidence Synthesis Working Group Lead, Director of a BEME International Collaborating Centre, and Director of Medical Education (Primary Care and Community) at University College London Medical School, London, UK.

Mohan Pammi

Mohan Pammi, MD, PhD, is a Director of the BEME International Collaborating Centre at Texas Children's Hospital, Director of the Evidence Academy at Texas Children's Hospital and Associate Professor, Baylor College of Medicine, Houston, Texas, USA.

Sebastian Alston

Sebastian Alston, MD, is a Director of BEME International Collaborating Centre and a Professor of Pathology at the Alabama College of Osteopathic Medicine, Dothan, Alabama, USA.

Deena Khamees

Deena Khamees, MD, is a Clinical Lecturer and Assistant Program Director of Emergency Medicine, Michigan Medicine, Ann Arbor, Michigan, USA.

William Peterson

William Peterson, MD, is a Director of Emergency Medicine Residency Preparatory Course, Assistant Clerkship Director, and an Assistant Professor of Emergency Medicine, Department of Emergency Medicine, University of Michigan Medical School, Ann Arbor, Michigan, USA.

Jennifer Stojan

Jennifer Stojan, MD, MHPE, is an Associate Professor of Internal Medicine and Pediatrics at the University of Michigan Medical School in Ann Arbor, Michigan, USA.

Cameron Pawlik

Cameron Pawlik, BS, is a first-year medical student at the University of Michigan Medical School, Ann Arbor, Michigan, USA.

Ahmad Hider

Ahmad Hider, MPhil, is a second-year medical student at the University of Michigan Medical School, Ann Arbor, Michigan, USA.

Michelle Daniel

Michelle Daniel, MD, MHPE, is Chair of the BEME Review Committee, Associate Editor for Medical Teacher, and Vice Dean for Medical Education and Clinical Professor of Emergency Medicine at the University of California, San Diego School of Medicine, La Jolla, California, USA.