Learnings from the forced transition of an industry supported educational programme for young experts in urology and oncology from face-to-face to digital during the COVID-19 pandemic

ABSTRACT

When the COVID-19 pandemic caused face-to-face meetings to be cancelled, an industry-sponsored educational programme, designed to develop skills and expand knowledge of young experts in oncology and urology, was forced to partially move from face-to-face setting to virtual meetings. In our outcomes analysis, we aimed to better understand what drives behavioural change following a series of educational interventions based on the physical or virtual formats. Therefore, we performed a structured outcomes evaluation for each educational intervention, including the perspectives of the learner and the teaching faculty. Our main findings were that “relevance” is the strongest driver of recall, satisfaction and behavioural change. Social interactions amongst learners and between faculty and learners are possible in the digital world, and we observed a trend of the young learners in favour of digital learning, especially with improved technical platforms enabling social interaction. Other findings were that new skills are required by the teaching faculty and that hybrid formats were identified by all participants as the model of the future. When developing future educational programmes, these specific needs of learners and faculty need to be considered and offer opportunities to develop more personalised programmes in order to increase learning impact.

KEYWORDS:

• Medical education
• lifelong learning
• digital formats
• digitalisation
• relevance of learning
• hybrid meetings

Introduction

Digitalisation in medical education started 35 years ago, but it took a pandemic to fundamentally challenge the way we provide medical education to learners worldwide. According to the literature, the first “virtual medical meeting” took place in 1986 [1], the first virtual congress in 1994 [2]. It took another 5 years until the first virtual medical meetings received Continuing Medical Education (CME) credits. The benefits of virtual meetings were identified early on [2]. Still, the disadvantages, especially the lack of social interaction and networking [3], led to a slow uptake of virtual meetings. However, when the COVID-19 pandemic caused face-to-face activity to be cancelled or postponed, it led to a forced digital transformation in medical education.

When assessing the strengths and barriers of virtual meetings and online learning, current literature focuses on practical barriers, such as internet connectivity and utility infrastructure [4,5]. Apart from hardware difficulties, Tabatabai et al. argue that maintaining standards in delivering educational content and accessing student learning can be extremely challenging when moving from face-to-face to digital learning [6]. Despite the increased number of publications in this area, knowledge about the efficacy of webinars is still considered limited and discussed controversially [7] or only explored through self-report questionnaires [8] offering limited insight into the reasons why.

Furthermore, the current literature on medical education under COVID-19 remains largely descriptive and limited to learners’ satisfaction and some knowledge gain. It provides only limited insight into the theoretical underpinnings of professional development and medical education and its subsequent application of knowledge in the context of COVID-19, with the majority of studies only offering a single actor perspective: either students or teachers [9–12].

The Project

The educational programme was implemented over the course of the year 2020 and was supported by Janssen Medical Affairs. Forced by the COVID-19 pandemic, the programme, originally planned as a series of face-to-face meetings, had to transition from a face-to-face (first meeting in January 2020) to a digital learning environment. The first digital meeting took place in July 2020, using Zoom as a platform. The second meeting in November 2020 applied to a more elaborate interactive platform (E-vade). It was designed to develop the skills and expand the knowledge of young experts in oncology and urology in the therapeutic area of prostate cancer (Table 1) relevant for their future role as faculty and educators. While the first face-to-face meeting enrolled a larger group of young experts, the digital meetings accommodated smaller learner groups to ensure interactivity in the digital space.

Table 1. Summary of the annual programme in relation to the three meetings

Meeting	Format	Aims
Meeting 1 (16 & 17 January 2020)	Face-to-face	Improving skills in the following areas: - Enhancing the ability to understand statistical data - Increasing presentation skills - Ability to develop and implement educational programmes
Meeting 2 (13 July 2020)	Digital	Improving skills in the following areas: - Identifying and prioritising educational knowledge gaps covered by educational programmes for young Oncologists and Urologists in the future - Identifying digital formats best suited to address knowledge gaps - Identifying faculty for future educational meetings
Meeting 3 (20 & 27 November 2020)	Digital	Discussing new data with impact on daily practice to improve knowledge and drive behavioural change by future educational programmes for young clinicians treating prostate cancer - Identifying new scientific data in PC with impact in daily clinical practice - Identifying and evaluating further educational knowledge gaps

Following completion of the educational programme, we performed a structured outcomes assessment to understand what drives learning impact and behavioural change in general and specifically what role does the format play.

As we aimed to analyse the potential differences between the setups, young experts (in the following referred to as “learners”) were only eligible for the assessment if they attended all three meetings (Table 2). These interviews were complemented by interviews with the teaching faculty (in the following referred to as “faculty”) that attended the face-to-face meeting and at least one virtual meeting. The rationale for including faculty members were i) to obtain reflections on learning outcomes from their perspective, and ii) to gain feedback on the programme design.

Table 2. Overview of participants across meetings and interviews

Who	Young Experts	Teaching Faculty/Advisors
Description	Clinical specialists in prostate cancer (future faculty)	Senior experts in urology & oncology/ Scientific committee/Teaching Faculty
Meeting 1	n = 33	n = 5
Meeting 2	n = 14	n = 2
Meeting 3	n = 21	n = 5
Interviews	n = 9	n = 4*

*one participant moved from the learner group to the senior faculty. Input was allocated accordingly

The interviews were implemented using a structured questionnaire consisting of a set of questions specific to each meeting and a learning objective. This was complemented by a set of nine questions specifically designed to gain insights about face-to-face versus digital meetings.

We applied two conceptual frameworks to develop the questionnaires in order to evaluate the outcomes achieved with the programme. With the first model, developed by Moore [13] and colleagues, outcome achievements were evaluated. As patient health and community health status as well as performance were out of the scope of the research, focus was on the central levels of the Moore model, related to learning: i) Moore Level 2: satisfaction; ii) Moore Level 3: knowledge; and iii) Moore Level 4: competence. Subjective assessments of the learners were combined with insights from the faculty and staff with regard to these levels.

The COM-B model was used to deepen our understanding of behaviour changes across the different levels, as well as the barriers that hindered behavioural change [14,15]. The model posits that all human behaviour is an interacting system in which Capability (physical & psychological), Opportunities (social & physical) & Motivations (automatic & reflective) interact to generate Behaviour (COM-B) [14,15].

Following institutional ethical approval, we used a qualitative research methodology, through in-depth structured interviews between March and May 2021, 4 months after the last meeting to ensure an appropriate time from the learning interventions to measure long-term changes. Interviews were conducted over Zoom, audio-recorded and transcribed verbatim. Data were analysed using NVivo software, and a process of investigator triangulation [16] was carried out.

Key Finding #1: Relevance Is Key

The concept of relevance, combined with the value of high engagement and interactivity, stands out as motivation for knowledge gain and behavioural change, while the format only played a minor role as a motivator or barrier.

Where learners did not see the relevance of a topic, they reported less satisfaction with the session and struggled with long-term recall. We further saw in the comparison of the two digital meetings that the content of the second digital meeting (focus on implementation of new clinical findings into clinical practice) was reported to be more relevant than the first digital meeting (focus on educational gaps and needs); but also the technology of the second digital meeting improved and was more interactive. This finding is supported by Hanke and colleagues [17], who suggested that future academic medicine programmes may need to incorporate new digital platforms besides other digital components.

Key Finding #2: Social Interaction Is Possible and Important for Learners in the Digital Environment

A prominent finding in the literature is that face-to-face meetings allow for more social interaction [3]. This was also reported by several of our learners in this study. Especially during the COVID-19 pandemic, the lack of social interaction has been identified as one of the elements limiting acceptance, satisfaction and success of virtual offerings.

However, personal attitudes and personalities may lead to different outcomes as our findings showed that especially learners who described themselves as introverted commented on how the digital platform allowed them to voice their opinion better and more often. Indeed, virtual settings and interactive digital formats including all participants can help those who are more introverted [18]. In our project, we saw that especially the technically improved platform of the second digital meeting, as well as interactive and engaging functionalities, was identified by the learners as opportunities for interaction and engagement among participants.

Key Finding #3: New Skills are Required for Faculty in the Digital Environment

Previous research has shown that faculty do need relevant up-to-date training in digital skills in order to successfully deliver digital training [19]. Our assessment confirmed previous findings that for senior faculty members the change from the physically known teaching environment to the digital setting is challenging mainly due to technology skills but also due to the fact that they cannot observe the learners as well as in the physical setting. Different teaching skills are needed.

Key Finding #4: Hybrid Approaches as the Preferred Model of the Future

Being asked “what do you expect to be the format of the future for educational programmes?”, the majority of learners indicated high satisfaction with the digital platforms and envision hybrid and/or blended approaches in future. In contrast to this finding, the faculty, while also expecting hybrid formats to become the format of the future, expressed a preference for face-to-face meetings, still experiencing the digital environment as a barrier and indicating that it negatively impacts on their teaching and requires new skills. The preference for hybrid format by the young experts reveals another interesting aspect. While virtual meetings have historically been considered as a replacement of face-to-face meeting, young experts consider online formats as an alternative format, equivalent to face-to-face meetings. This is supported by Hameed and colleagues [20] who report that in the urological community hybrid meetings are likely to be the next logical step for future CME meetings.

For detailed results, please refer to Table 3 (outcomes per learning objective) and Table 4 (quotes per key finding).

Table 3. Outcomes per learning objective

Objectives	Findings	Comments
LO 1: Increase ability to understand statistics and apply in a professional environment	High satisfaction Learners confirm increased ability Learners confirm that they applied this new skill	The subjective perception of the learner was confirmed by the perception of the faculty (increased ability) and by staff (application of new skills in local meetings)
LO 2: Improve presentation skills and performance as a speaker	High satisfaction Learners confirmed increased knowledge on how to present Learners confirmed that they applied the new skill in meetings (online and offline)	The subjective perception of the learner was confirmed by the perception of the faculty (increased ability) and by staff (application of new skills in local meetings) The learners indicated that they were able to apply the new skill in a virtual setting
LO 3: Increase ability to develop an educational programme	Medium satisfaction and recall of the sessions Most learners confirmed increased knowledge of how to do it Most learners did not apply this new skill in real life	The learners indicated that this topic was less relevant for them Several learners indicated that this is not part of their role in real life
LO 4: Increase ability to identify knowledge gaps	Medium to high satisfaction and recall of the session Some learners did not really understand the purpose of the session Most learners did not apply this new skill in real life	The learners indicated that they had good and stimulating discussions
LO 5: Increase ability to identify and allocate digital channels	Low recall of the session Some learners did not really understand the purpose of the session	The learners indicated that this topic was less relevant for them Several learners indicated that this is not part of their role in real life
LO 6: Identifying faculty for future educational meetings	Medium recall of the session
LO 7: Identify new scientific data in PC with impact in daily clinical practice	High satisfaction and recall of the session Most learners confirmed increased awareness and knowledge on how to do it	Learners indicated that the topic was relevant, the session was engaging and interactive Perception was confirmed by faculty and staff

Table 4. Supporting quotes for key findings 1–4

Key Finding	Quotes
Key Finding #1: Relevance is key	“ … it’s such a frequent problem that we face”.. (learner) “… I remember about all those parts about maybe the the one that impacted me more, or I remember it more because it’s very linked to our practice … “ (learner) “ it was very relevant, especially for the ones I know … .(staff)
Key Finding #2: Social interaction is possible and important for learners in the digital environment	“When we meet in person, we mostly don’t talk. And in digital in the last meeting, we talk a lot by WhatsApp by mail. So I don’t think digital is a barrier .” (learner) I“t was extremely useful, because it was a real moment of an interaction, of interface” (learner) “… because in the face-to- face meeting, I think I didn’t participate a lot because I was a bit ashamed …And in the digital, you know, you don’t feel like that … “ (learner) “the breakout rooms in the online format worked really well” (learner)
Key Finding #3: New skills are required for faculty in the digital environment	“I’m not a user of, of these kinds of tools … if you want to interact more than 20 simply forget it” (faculty) “it’s quite hard to see when people are engaging … it’s still quite hard to have a proper discussion” (faculty) “I’m not a digital guy for the new tools …I’m not using them” (faculty) „these guys (= the young experts) really know what they’re talking about. So I think it was the, you know, the student teaching the teachers on that one“ (faculty)
Key Finding #4: Hybrid approaches as the preferred model of the future	“I would go for a combination, if it’s possible also for the future … ” (learner) “It has been a mixture of both things. Face to face is dated, but virtual is also, well, it’s difficult, it’s very difficult to have a five days face to face meeting, its more feasible to have a two day’s face to face meeting and the equivalent in terms of time of the three extra days by one or two hour sessions virtual, either prerecorded or better interactive, it can be both pre recorded and/or interactive, so it has to be it has to be both, it will be both. I do believe it will be both”. (faculty) “I think we have to organise it to, in one event clinicians can go to the live meeting and digital meeting” (staff)

Conclusion

The concept of relevance stands out as a main driver for recall, satisfaction, knowledge, and long-term competence gain, while the format (face-to-face or digital) was identified as playing a minor role for learning impact.

We further saw a trend in the young learners’ group in favour of digital learning, as improved technical platforms seem to enable social interaction also in a digital environment. Independent of personal preferences and capabilities, all stakeholder groups expected hybrid models of face-to-face and digital elements to be the future model.

Nevertheless, technological barriers in medical education have been reported as problematic [4], and we encourage future work to look at how individuals can be supported with technology, particularly in a digital climate. This will also aid in the implementation of new skills for faculty and learners in the digital field.

We could see that even a small group of learners and faculty confirm findings and hypotheses from other authors. We identified specific needs from learners and faculty in the digital environment, which we plan to implement in future educational activities. Therefore, our project may serve to inspire researchers to strive for confirmation in a larger setting.

Acknowledgments

This outcomes evaluation was conducted and led by LLH Concepts. We would like to thank our research team for the dedicated support they provided: Tong Li and Trishna Chauhan (Senior Researchers, LLH Concepts). We thank all other members of the working group who are not co-authors but contributed to the discussion of the results.

Disclosure Statement

IW is president and founder of LLH Concepts. She has disclosed that she has worked as a consultant to Janssen on the programme. IV is professor of behavioural science at the University of Warwick and head of research at LLH Concepts. He declares no relevant financial or non-financial competing interests. ET is lead of external scientific relations EMEA at Janssen Cilag Pharma GmbH. She was involved in the design and conduct of the study and preparation and final approval of the manuscript, but she was not involved in the collection, management, analysis, or interpretation of the data.

Additional information

Funding

The educational program and outcome research was funded by Janssen, The Pharmaceutical Companies of Johnson & Johnson.