Outcomes and Observations of On-line CME Activities during the Pandemic

ABSTRACT

The COVID-19 pandemic created an environment where the majority of continuing medical education (CME) and continuing professional development (CPD) activities needed to be delivered digitally. Producing digital materials for 16 separate learning activities (four learning journeys for each of four topic areas) in 2021 provided challenges and raised points of interest and discussion for a small, Italy-based provider of CME and CPD. This study presents outcome metrics from four live, interactive webinars. A variety of promotional efforts, including the strategic use of social media, generated interest and participation; feedback from the European Accreditation Council for Continuing Medical Education standard questionnaire to participants provided rates of satisfaction; subject knowledge and self-reported competence was measured by responses to pre- and post-event and follow-up (after 3 months) questionnaires. Post-event analysis of processes prompted introspection on the learning journey outcomes and methods of analysis. This paper discusses these observations, including potential innovations for future activities (e.g. reconfiguring the e-learning platform to capture time spent on learning activities), and also discusses issues in learner behaviour that impact CME provision and evaluation.

KEYWORDS:

• Outcomes
• continuing medical education
• continuing professional development
• digital learning
• participation
• knowledge
• competence

Introduction

Working in health care requires a lifelong commitment to learning. It is vital to stay abreast of best practices, to assimilate innovations and recommendations from guidelines or expert opinions, and to discontinue ineffective practices [1]. Continuous learning is so important for healthcare quality that in many countries, continuing medical education (CME) is a mandatory component of physician licencing requirements [2] and to maintain certification; participation can influence promotions and career progression, and physicians can be penalised financially or be officially reprimanded for lack of involvement [3]. CME content is mainly produced by medical societies, educational institutions, and independent CME providers receiving independent educational grants. Accreditation and quality control of activities are often the jurisdiction of country-level medical authorities. In Europe, pan-European CME credits can also be provided by the European Accreditation Council for Continuing Medical Education (EACCME®), although acceptance of these credits varies by speciality and geography, they are limited to only some specialities and are not valid for any general practice-based CME (EACCME: https://eaccme.uems.eu/home.aspx). While digital CME is available to learners globally via the internet, accrediting systems (i.e. quality control systems) are a product of national policy and context. An international organisation, the International Academy for CPD Accreditation (IACPDA), is working to converge on accrediting principles and processes [4].

This paper describes a series of CME activities that took place in 2021 and were developed by one provider, the Scientific Seminars International Foundation, based in Rome, Italy. This paper explores how medical professionals engaged with CME activities and resulting educational impact. All activities were digital because of the limitations on live in-person meetings resulting from the COVID-19 pandemic, and all materials were hosted and freely accessible on an e-learning platform (https:// cme-learning.scientificseminars.com/).

CME Activities

The CME activities covered the following medical speciality areas: endocrinology (thyroid disorders, growth disorders, and diabetes/pre-diabetes) and hypertension/cardiovascular disease. In 2020, multi-modal need assessments were conducted for each speciality area to identify knowledge and professional practice gaps. From this, and with input from a medical advisor, the experience of the Scientific Seminars team and accumulated feedback and interaction with active learners, a digital educational journey for physicians was created for each therapeutic area (Table 1). The activities were designed as standalone educational interventions, thus not requiring participation in the full learning journey, but maximum impact was achieved if the whole journey was viewed. Ultimately, each learning journey included a 15–20-min video interview with an opinion leader on a topical subject; an interactive clinical case presentation, with the faculty asking provocative questions throughout for the learners to answer in relation to the next diagnostic steps or treatment; a written report by an opinion leader published on the Scientific Seminars webpage from a congress or annual meeting of a medical society specialising in the topic of interest (e.g. the European Association for the Study of Diabetes [EASD] annual meeting); and a live webinar with presentations by two faculties with moderation by a third. There was no preferred order for learners to view the activities, and components of each journey were released at least a month apart. Participation numbers for each learning journey were greatest for the webinar, so webinar figures have been used in this report to demonstrate learning outcomes, learner behaviour and interaction. The webinars were all of one-hour duration and were all accredited by EACCME® for one credit each.

Table 1. Activities released for each digital-learning journey during 2021.

Specialist area	Activity	Title	Release month
Diabetes/pre-diabetes	Video interview	Metabolic health does matter: Lessons from the COVID-19 pandemic	April
	Live webinar	Diagnosis of diabetes in ethnic minorities: Challenges and strategies to bridge the care gap	July
	Interactive clinical case	Type 2 diabetes in adolescents: Differential diagnosis and therapeutic algorithm	September
	Congress report	Virtual European Association for the Study of Diabetes annual meeting	November
Growth disorders	Video interview	Transition of care for girls with Turner syndrome	May
	Live webinar	Prader-Willi syndrome: Challenges and best practice solutions for each life stage	June
	Interactive clinical case	Advances in early diagnosis of congenital hypopituitarism	December
	Congress report	European Society of Paediatric Endocrinology annual congress	October
Hypertension/cardiovascular disease	Video interview	Stable chronic angina: Revascularization versus medical therapy	June
	Live webinar	Hypertension and beyond	November
	Interactive clinical case	Clinical management of hypertension associated with high cardiovascular risk	June
	Congress report	Virtual European Society of Hypertension/International Society of Hypertension congress	May
Thyroid disorders	Video interview	Iodine deficiency	April
	Live webinar	Fertility issues related to thyroid disorders	September
	Interactive clinical case	Thyroid hormones and lipid metabolism	October
	Congress report	Latin American Thyroid Association congress	December

Learner Recruitment and Data Collection

Given that there were no live events during the pandemic, activities were primarily promoted on social media. Metrics from Facebook, LinkedIn and Twitter were collected to provide overall figures for the reach of posts and for the number of people who interacted with the activities (i.e. likes, shares, video views, comments and link clicks). Details of the events were posted on Scientific Seminars social media sites, and interested health-care professionals were encouraged to join in for further information on freely available learning events and to share details with their colleagues. Additional health-care professionals were reached by direct email, advertising through national medical societies and in medical journals and direct contact with larger hospitals and universities.

When learners started any activity, they were asked to provide demographic information (geographical location, profession, speciality area and the number of years working in the speciality area [<5 years, 5–10 years, >10 years]), but learners could engage with and watch the activities without providing information or completing the pre- and post-activity questionnaires. The voluntary participation analytics provided Moore’s level 1 data [5].

Learners who applied for EACCME® credits were required to evaluate the activity based on standardised criteria. The question “How useful for your professional activity did you find this event?” was used to determine learner’s level of satisfaction with the education (Moore’s level 2 data [5]).

The e-learning software used to deliver the educational activities did not capture the amount of time spent by a learner on an activity.

Measurement of Change in Knowledge and Competence

Learners were encouraged to complete a series of multiple-choice questions prior to beginning the webinars. These questions were directly linked to the learning objectives of the activities and were related to the content within the presentations. The same questions were asked of learners as soon as they completed the webinar. Measurements were simple – the percentage of correct pre-activity answers was compared with the percentage of correct post-activity answers, with no matching to see if those who provided incorrect answers initially, then provided correct answers post-event. The difference in the percentage of correct answers pre- and post-activity represented a relative measure of change in knowledge for those answering pre- and post-activity questions.

Learners were also asked before the webinars to rate their own level of competence in a clinical scenario from the activity – for example “Rate your competence in managing type 2 diabetes in adolescents”. Answers were chosen from “Excellent”, “Very good”, “Adequate”, “Needs improvement” or “Poor”. The same questions were asked 3 months after the activity (via email). The differences between pre-webinar and follow-up values were compared and provided a change in self-reported competence score for the activity. Similar to the measurement of change in knowledge, there was no matching of learner’s pre and post responses.

Measurement of the Impact of the Webinar

Immediately after completing the activity, learners were asked if the material they had viewed would change their future clinical practice, and if so, they were asked to describe the changes they were planning to make. These learners were contacted again after 3 months (via email) and asked if they had made the changes that they had intended to make.

Data Analysis

Simple analyses were used to compare the percentage of correct answers to the knowledge and competence questions before and after the activity and to compare changes in behaviour metrics. No statistical tests were used to determine the significance of any differences detected.

Results

Participant Numbers

Learner numbers across the four learning journeys are detailed in Table 2. For the webinars, demographic data were volunteered by 11,868 learners (~3000 per subject area), from 44 countries. A small number of learners opted to complete pre- and/or post-activity questionnaires for the webinars. Using the diabetes figures as an example, demographic data were provided by 2260 learners, 52 (2.3%) completed the pre-event questionnaire, 37 (1.6%) completed the post-event questionnaire and 15 (0.7%) completed the follow-up questionnaire (Table 3).

Table 2. Learner numbers for the four scientific seminars CME learning journeys.

Specialist area	Number of learners providing demographic data	Number of different countries learners were from	Number of learners who completed the EACCME® accreditation process
Diabetes/pre-diabetes	3260	44	5
Hypertension/cardiovascular disease	2624	14	18
Growth disorders	2035	32	22
Thyroid disorders	3949	25	39

Table 3. Results in changes in knowledge and self-assessed competence from webinar learners who completed the pre-/post-/follow-up questionnaires.

Subject	Change in knowledge	Self-assessed change in competence	Number of pre-webinar responses	Number of post-webinar responses	Number of follow-up responses
Diabetes/pre-diabetes	12.4%	50.8%	52	37	15
Hypertension/cardio vascular disease	12.6%	22.3%	56	18	15
Growth disorders	5.4%	58.2%	89	52	15
Thyroid disorders	10.3%	20.2%	63	39	23

Of the learners who answered the pre-activity questionnaire, most were physicians (thyroid disorders: 63 [51%]; growth disorders: 89 [64%]; diabetes/pre-diabetes: 52 [77%]; hypertension/cardiovascular disease: 56 [82%]).

Event Evaluation

Learners who applied for EACCME® credits were required to evaluate the activity based on standardised criteria. The webinars each scored highly on all standardised criteria, but particularly on the quality of the faculty, the panel discussion, overall organisation and usefulness of the webinars (Figures 1 and 2). The overall satisfaction rate was high for each of the webinars (Figure 2A).

Figure 1. Scores for aspects of the webinars given by learners who applied for EACCME® credits. Questions are answered using one of two 5-point scales – very much, somewhat, not much, undecided, not at all or excellent, good, fairly good, poor, very poor.

Figures plotted are the percentage of learners who rated the aspect of the activity as either Very much/Somewhat or Excellent/Good.

Figure 2. Learner feedback from the webinars: (A) Satisfaction rate (from EACCME standardised evaluation). (B) Self-reported intention to change practice. (C) Self-reported actual change in clinical practice.

Learning Outcomes

Each webinar led to relative increases in both knowledge and self-reported competence for learners, based on questionnaire responses (Table 3). However, the number of learners who provided feedback was low, and progressively decreased from the pre-event total.

Learner Reported Changes in Behaviour

A high proportion of learners on each journey who responded to the post-activity survey immediately after the webinar committed to making changes to their clinical practice (Figure 2B), and the majority who completed the follow-up survey stated that they had enacted these changes (Figure 2C). However, numbers were low, and there was no tracking to follow individual learner responses.

Discussion

The four 2021 digital learning journeys on diabetes/pre-diabetes, hypertension/cardiovascular disease, growth disorders and thyroid disorders were effective CME activities. They were successfully delivered during the challenging circumstances of the COVID-19 pandemic, reaching high numbers of healthcare professionals, mainly physicians. Utilising social media heightened the profile of the activities. Satisfaction rates were high for learners who provided information, and the activities produced increases in knowledge and self-reported competence in learners who provided feedback (measured using simple calculations of percentage increases).

Simulescu et al. (2022) reported that moving activities to digital-only formats in 2021 had the effect of increasing the audiences of online CME activities versus in-person events [1]. Flexible participation, with learners able to join activities from their own countries and with others in the same time zone, as well as lower cost and time barriers compared with in-person attendance, facilitated greater access to education [1]. Online lecturers can contribute and present their content globally, making current learning available for attendees without the need for either party to travel [6].

Collectively, the webinars described in this paper attracted learners from 44 different countries. The activities were developed following research on physician needs and practice gaps but highlight an interesting aspect of worldwide CMEs: CME events, particularly those online, finds relevance and/or audience beyond who they were strictly developed for. Thus, metrics for assessing outcomes may be less effective or may produce unexpected results.

Although digital education and e-learning have been a part of the CME armamentarium for more than 20 years, the ability to use these platforms more widely during the COVID-19 pandemic provided an opportunity to innovate – and retain – lifelong learning for healthcare professionals, without interruption. However, CME activities aim to provide more than straightforward learning experiences, and the digital environment typically gives limited scope for collegial interaction, debate and discussion, networking and general cross-fertilisation of ideas, compared with in-person events. Both digital and in-person activities have their merits, and a good balance of both is likely a useful way forwards, as is the consideration of the hybrid learning environment, combining both in-person events with synchronous digital components. Comparing the benefits between e-learning and traditional learning techniques, a meta-analysis of 11 randomised clinical trials involving 2491 nurses and student nurses (conducted before the pandemic) found no statistical difference in participants’ knowledge, skills and personal satisfaction between the learning processes [7]. A questionnaire-based study on the preferred way to receive CME completed by 2949 German-speaking healthcare professionals (mainly physicians [78.3%]) found the majority of participants (87%) wanted online CME to continue in a similar manner post-pandemic as during the pandemic, with 30% of respondents stating that they would rather only participate in online CME, 14% expressing a preference to only participate in face-to-face CME events and 56% a combination of both face-to-face and online CME formats [6].

CME providers generate outcome data for each activity that they develop; they use these data to assess the quality, educational and practical impact of the event, webinar or article so that future activities can evolve. Outcome data generation traditionally involves assessing the number of learners reached and the impact of the activity on these people. However, in general, learners choose not to provide information – either simple demographic data or pre-/post-event questionnaires – as shown in the current study (Table 2). The reasons for this reluctance to complete (or provide feedback) have not been widely studied but may be related to barriers to digital competence and high workload [8].

Such a reluctance to engage with post-event questionnaires identifies a potential problem in CME provision: good-quality feedback is needed from events to help shape future activities, and to show the value of the offering to faculty members and potential grantors, yet such data are very difficult to obtain. In addition, comparing answers from ever-lower overall numbers of survey responses minimises the impact of the findings, as does the fact that different learners may participate in pre- and post-event questions.

Limitations

Analysing this dataset, several points became apparent:

• No record was made of the time spent by a learner on an activity, all were assumed to have completed the activity. Practically, it is extremely rare for there to be 100% retention of participants in any form of educational activity, anecdotally, the more typical retention rates are ~30%.

• These worldwide CME events are presented in English but are accessed by great many learners whose first language is not English.

• There was no matching of learners providing pre- and post-activity questions. Therefore, pre/post-comparisons are relative and not absolute.

Innovations envisioned to counter these shortcomings include reconfiguring the e-learning platform to capture the time spent on each activity and matching learner data to pre-, post- and follow-up questionnaires to establish accurate outcome data. The possibility of tracking the number of activities that individual learners engage with, and mapping this information to their knowledge and competency outcomes would also be worthwhile. The questions of whether reliance on English language and digital competence are clouding participation in (and generating outcome data from) CME activities is intriguing and worthy of further research.

Conclusions

Throughout 2021, CME providers and faculty were challenged to provide educational activities to healthcare professionals in a wholly digital format. The digital learning journeys described met this challenge and used a strong social media presence to provide good levels of event awareness. Online participants were very satisfied with the digital learning journeys: those who completed post-event questionnaires reported knowledge and practice changes in line with the learning objectives of each activity. Despite these positive findings, the numbers of responders to feedback requests for CME digital events were low, which limits the quality of analyses of event impact.

Significance

This paper reports on the outcomes achieved with a series of global CME webinars, covering four different therapeutic areas, provided free of charge to physicians during 2021, and highlights some challenges facing providers in terms of participant engagement and feedback data, together with unexpected positive outcomes (particularly the reach of activities to 44 countries).

Acknowledgments

The authors thank Linda Edmondson and Rebecca Mant (UK-based freelance medical writers) for editorial assistance and Flaminia Masprone (Employed by the Scientific Seminars International Foundation at the time of the activities) for management and support of the learning journeys and outcome metrics.

Disclosure Statement

The activities described were funded through independent educational grants received from Merck Healthcare KGaA, Darmstadt, Germany.

Additional information

Funding

This work was supported by independent educational grants received from Merck Healthcare KGaA, Darmstadt, Germany.