Personalised versus non-individualised case-based CME: A randomised pilot study

ABSTRACT

The PinPoint Case Platform (PPCP) offers independent online case-based CME. To align with personal learning needs, a functionality of needs assessments (“QuickScan”) was developed, directing users to follow personalised case journeys. A randomised study was conducted, comparing its effectiveness, time efficiency and user experience with a format of non-individualised case-based learning. Forty-two residents in urology from five European countries were randomly assigned to follow non-individualised case-based learning (control group) or a needs assessment plus personalised case journeys on different topics in prostate cancer. After performing a pre- and post-assessment, both groups showed a similar increase in test scores (Mann-Whitney U = 247; p = .113), but the time needed for completing the learning exercise was significantly lower in the group with the personalised approach (median: 45 vs 90 minutes; Mann-Whitney U = 97.5; p = .0141). The quality of the two learning methods was similarly well received by both groups. In conclusion, learners who followed personalised case journeys learned similarly effective but more time efficient than non-individualised case-based learners. Future studies should determine if these findings can be extrapolated to board-certified physicians following CME activities.

KEYWORDS:

• Continuing medical education
• online CME
• case-based learning
• individualised learning
• prostate cancer

Introduction

Continuing medical education (CME) should aid healthcare professionals (HCPs) to apply new scientific evidence into clinical practice with the goal to improve clinical decision making and eventually also patient health outcomes. The steadily increasing volume of literature to assimilate puts the time available for CME under pressure. Because traditional CME formats (e.g. lectures and self-study of scientific literature) often fail to change the clinical decision performance of physicians, many CME providers are directing their efforts towards the development of more interactive and competency-based educational strategies [1–3]. Especially during times of the COVID-19 pandemic and growing environmental awareness, digitisation of CME has become increasingly relevant [4,5]. Compared with traditional learning formats, online CME is more easily accessible, time efficient, self-paced and cost saving [6]. Besides these advantages, several studies have demonstrated that online CME can exert significant improvements in knowledge and clinical practice [7–10]. However, digitisation of educational materials as such is not the key to success. Online educational materials should be short, highly relevant and patient-focussed [11–13]. These aspects may all be achieved by offering online CME in a case-based format with educational content that is ideally tailored to the specific needs of the individual learner. Although needs assessments are becoming well integrated into CME programmes, they mostly focus on educational needs at an aggregate level. Nevertheless, greater performance may be achieved when online CME addresses individual gaps in knowledge, skills and practice performance [11,12,14].

Based on the above-described insights and premises, the Mirrors of Medicine™ PinPoint Case Platform (PPCP) (https://ppcp.mirrorsmed.org/) was developed and launched in 2018 (Table 1). It is the first online case-based learning platform to receive accreditation by the European Union of Medical Specialists – European Accreditation Council for Continuing Medical Education (UEMS-EACCME). The PPCP provides independent online CME with free access to students, residents and healthcare professionals. The content development is supported by multiple unrestricted educational grants. Recently, a new feature was added to the platform allowing users to follow personalised case journeys (PCJs) directed by the outcomes of individual needs assessments (QuickScans) (Table 1). To understand the potential benefits of adding this feature to the PPCP, we conducted a randomised study comparing the effectiveness, time efficiency and user satisfaction of non-individualised case-based learning with learning through PCJs addressing individual learning needs identified by QuickScans.

Table 1. Characteristics and functionalities of the PinPoint Case Platform (PPCP).

Cases and topics Cases are structured by disease area and topic (delineated subject of interest) Each topic contains at least 5 cases Each case contains a short patient description with multiple clinical options After choosing an option, the following feedback is provided: Scores of peers who answered the same question Validated recommendation of an international multidisciplinary expert panel Short summary of relevant literature and guidelines related to that case

Development and validation of materials Topics are selected and defined by the Scientific Committee based on educational needs in a specific disease area Cases and feedback are prepared and reviewed by physicians involved in clinical decision making in the topic under consideration Clinical options within cases are validated by a multidisciplinary expert panel (6–12 members), using the RAND/UCLA Appropriateness Method [15]. The extent of appropriateness of a clinical option is expressed on a 9-point scale going from inappropriate (1–3) to equivocal/uncertain (4–6) to appropriate (7–9) Coaching, support and coordination is managed by an independent CME provider

Basic functionalities Users can view and study cases and topics in any order on the platform Users can subscribe to a (bi)weekly case programme via email in any of the disease areas Each topic provides an accredited CME course, consisting of 5 consecutive cases with at least one what-if scenario, and followed by an assessment of 10 questions on the cases and related evidence

Personalised case-based learning To assess individual learning needs, “QuickScans” per disease area and speciality are available. These are selected assessment questions, considered most relevant by at least three experts per speciality After answering each question included in the QuickScan, learners receive direct feedback (correct answer, overview of supporting evidence and comparison with the scores of peers) Based on the outcomes of the QuickScan, learners are directed to follow “Personalised Case Journeys” (PCJs), consisting of an automated flow of tailored cases and questions, again with direct feedback In the PCJs, questions with wrong answers are repeated at the end of a series until all questions are answered correctly PCJs can be completed at once or in stages via an email programme

PCJs: Personalised Case Journeys

Materials and Methods

Study Population

The study population consisted of residents in urology (all study years) from five European countries invited by their supervisors who contribute as experts to the PPCP. The choice for residents was made to facilitate recruitment of a homogeneous study population. Participants needed to be unfamiliar with the platform and willing not to use the platform outside the context of the study. In addition, they were instructed not to use information from any other educational sources during the conduct of the study and participated voluntarily to the study after providing written informed consent.

Study Design

Participants were block randomised to follow non-individualised case-based learning (control group) or a personalised approach consisting of a QuickScan plus PCJs (QS-PCJ group) (Figure 1). Before learning on the PPCP, participants of the control group had to complete an electronic survey, consisting of 27 assessment questions that were derived from the relevance ranking lists generated during the development of the QuickScans. The participants of the QS-PCJ group were requested to complete the same 27 assessment questions but on the PPCP in the form of a QuickScan, providing targeted feedback by showing the correct answer with supporting evidence. After completion of the pre-test, the control group was invited to study five topics on the PPCP, while the QS-PCJ group was requested to complete the selected PCJs based on the individual learning needs identified through the QuickScan. Ten days after the start, both groups received an electronic survey, including the same 27 pre-assessment questions but in a different order. Participants were also requested to report the time spent to study the five topics (control group) or to document how much time was needed to complete both the QuickScan and PCJs (QS-PCJ group). To evaluate user experience, participants of both groups were asked to assess different aspects of the PPCP on a 9-point Likert scale.

Figure 1. Overview of the study design. The same 27 pre-assessment questions were included in the electronic survey and QuickScan but, in contrast to the electronic survey, the QuickScan provided targeted feedback (correct answer with supporting evidence).

Statistical Analysis

Descriptive variables are summarised as medians with interquartile ranges. The test scores on the pre- and post-assessment were calculated as percentages of the maximum score (maximum score = 27). Given the nature of the data, we did not assume a normal distribution and non-parametric statistical tests (Wilcoxon test and Mann-Whitney U test) were used. All statistical analyses were performed using the IBM SPSS Statistics 28 software package (SPSS Inc.).

Results

Study Population

The study population consisted of 42 junior and senior residents in urology from different centres across five European countries. Out of the 42 enrolled residents, 39 completed both the pre- and post-assessment. Participants with incomplete data (n = 3) were excluded from the analysis. Basic characteristics of the population by study arm are shown in Table 2. Statistical analyses, where possible due to the low numbers, did not reveal any significant differences between the two study groups.

Table 2. Description of the study population.

	Control (n = 19)	QS-PCJ (n = 20)	Total (n = 39)
Country
Belgium	5	3	8
The Netherlands	0	2	2
Slovenia	1	2	3
Turkey	10	10	20
UK	3	3	6
Years of residency
1–4	13	15	28
5–6	6	5	11

Absolute numbers. QS-PCJ: QuickScan plus Personalised Case Journey

Learning Process

Participants from both groups completed the 27 questions of the pre-assessment or QuickScan. Following the pre-assessment, all residents in the control group went through the indicated five topics, including a total of 25 cases. In the QS-PCJ group, the number of PCJs to follow was dependent on the outcome of the QuickScan and varied between 1 and 5. Three-quarters had to follow at least four PCJs with all participants fully completing the individually assigned case journeys.

Learning Effects

After completing the non-individualised case-based learning activity on the PPCP, median test scores significantly improved from 66.7% to 81.5% (Z = 163; p = .006) in the control group (Figure 2(a)). A significant increase was also observed in the QS-PCJ group, showing an improvement in test scores from 74.1% to 92.6% (Z = 185; p < .0001) upon completion of the individually assigned case journeys (Figure 2(b)). Although the change in the percentage of correct answers was slightly higher in the QS-PCJ group, differences did not reach statistical significance (Mann-Whitney U = 247; p = .113) (Figure 2(c)).

Figure 2. Comparison of learning effects between the control and QS-PCJ group. (A) Pre- and post-test scores of the control group. (B) Pre- and post-test scores of the QS-PCJ group. (C) Change in the percentage of correct answers in both groups. The graphs represent the median values with interquartile ranges. QS-PCJ: QuickScan plus Personalised Case Journey.

Time Efficiency

The median time reported by the participants in the control group to study all five topics on the PPCP was 90 minutes. Even though one outlier was included in the control group, the self-reported learning time was significantly lower in the QS-PCJ group, reporting a median time of 45 minutes to complete both the QuickScan and PCJs (Mann-Whitney U = 97.5; p = .0141) (Figure 3).

Figure 3. Self-reported time spent with learning on the PPCP. The graph represents the median values with interquartile ranges. QS-PCJ: QuickScan plus Personalised Case Journey.

Learner Experience and Assessment of Quality Aspects

Both groups showed a high and similar appreciation of studying on the PPCP scoring a median of 8 on a 9-point scale for five different aspects (Table 3). Several other aspects on the quality of the platform received similar high scores without significant differences between the groups (Supplementary file 1). Additionally, participants of both groups were as likely to recommend the PPCP to a colleague.

Table 3. Perception of studying on the PPCP.

Aspect	Control (N = 19)	QS-PCJ (N = 20)
Helps me to better understand my learning needs in prostate cancer	8.0 [7.0–9.0]	8.0 [8.0–9.0]
Helps me to more effectively learn about prostate cancer	9.0 [7.0–9.0]	8.0 [7.3–9.0]
Helps me to increase my productivity in learning about prostate cancer	8.0 [7.0–9.0]	8.0 [7.0–9.0]
Makes it easier for me to learn about prostate cancer	8.0 [7.0–9.0]	8.0 [7.3–9.0]
Is an inspiring way of learning	8.0 [7.0–9.0]	8.0 [7.0–9.0]

Numbers are expressed using a 9-point scale: 1 = strongly disagree; 5 = neutral; 9 = strongly agree and represent the median values with interquartile ranges. QS-PCJ: QuickScan plus Personalised Case Journey.

Discussion

End of 2018, the PPCP was launched via a weekly case programme, offering interactive online case-based CME by easy-to-digest, self-directed and competency-based learning content that can be continuously updated to advancements in clinical practice. Three years after its launch, the PPCP has around 3,600 active users mainly in the field of uro-oncology [16]. Recently, a feature for allowing personalised learning on the PPCP has been developed, offering users the possibility to follow PCJs based on the outcomes of individual needs assessments (QuickScans). This feature is built on the premise that CME will accomplish the greatest performance gains when tailored to the needs of the individual learner. When reviewing learning needs, a distinction should be made between perceived and unknown learning needs [17]. Perceived learning needs, which are usually self-recognised, may not necessarily reflect a person’s “actual” learning needs [17]. Hence, it is not surprising that there is only a low correlation between self-assessments of physicians and their performance on objective knowledge tests [1]. An effective needs assessment should, therefore, aim to identify a learner’s unknown needs [1,17]. On the PPCP, this has recently been achieved by implementing QuickScans aiming to reveal true performance gaps that may have been unknown to the learner. Anticipating on individual performances and/or knowledge gaps was the target for the generation of adaptive PCJs fitting the need of every single learner focussing on true unperceived learning needs instead of educational learning needs that address a group of learners as a whole. By customising the learning pathways, the PCJs allow each learner to focus on individual gaps in a certain learning domain. This approach represents a simplified version of an adaptive e-learning strategy, known as “curriculum sequencing” [18,19]. In addition, the QS-PCJs on the PPCP are adapted to the speciality of the learner within a disease area, but to develop a more sophisticated, algorithmic approach within the framework of adaptive e-learning more data need to be collected that will allow further tailoring of the PCJs based on the knowledge level of the individual learner. Besides being adaptive, the PCJs integrate the concept of learning from mistakes by repeating incorrectly answered cases or questions until they are correctly answered. This concept is believed to further increase learning gains through emphasising discrepancies between the learner’s choice and the opinion of peers along with providing evidence to support the correct answer to ultimately induce changes in individual performances [20–22].

In this randomised pilot study, we compared the short-term learning effects of non-individualised case-based learning (control) with a personalised approach consisting of the QuickScan and related PCJs (QS-PCJ). The results showed that both learning methods significantly improved learning outcomes, and are, thus, effective in inducing short-term learning effects and potentially changes in clinical performance. Although improvement was higher in the group following PCJs, the difference in test scores between the two groups did not reach statistical significance, most likely due to the small group sizes. Despite this lack of statistical significance in improvement of test scores, participants following PCJs achieved similar short-term learning effects in significantly less time. It should be noted that the learning time reported by the QS-PCJ group included both the time spent on the QuickScan and PCJs, while the control group only reported the time needed to study the five topics on the PPCP. This represents the actual learning situation on the PPCP, where users have the possibility to either study topics based on their own interests or to conduct the QuickScan to focus on their individual learning needs. In addition, the QuickScans on the PPCP are shorter (maximum 10 questions), and therefore, the actual time efficiency of the personalised learning approach is probably underestimated in the current study. Regardless of the learning format that was followed by the participants, all aspects related to the perception of studying and quality of the content were very well received.

Although randomised, the herein reported study results are merely descriptive and hence, a follow-up randomised controlled trial should be designed to confirm the findings. In the current study , the limited number of participants most likely contributed to the relatively high degree of variation in the test scores. In addition, the time spent on the platform was self-reported by the participants, which could have biased to some extent the time efficiency results. For the purpose of this study, only residents were included because this facilitated (easy) recruitment of a homogeneous study population. Although senior residents had slightly higher scores on the pre-assessment than residents in the first years of their training, their distribution was equal between groups and did, therefore, not affect the study outcomes. However, as the PPCP is focussed on continuing medical education, residents are not the most representative of the target audience. In addition, they were recruited by their supervisors, which may have affected their incentive to participate and their performance on the learning activities. Therefore, additional studies should determine if the findings are also valid for board-certified physicians and other healthcare professionals.

Conclusions

To the best of our knowledge, the PPCP (https://ppcp.mirrorsmed.org/) is the first platform offering both non-individualised and personalised case-based learning. Both learning methods were shown to be effective in inducing short-term learning effects; however, following a tailored programme was significantly more time efficient without compromising the short-term learning outcomes. Because both learning methods were equally effective and similarly well received by the participants, the PPCP offers users the possibility to either choose for non-individualised or personalised case-based learning depending on personal preferences and interests, though mentioning time saving as the principle advantage of the tailored approach. Future trials will be initiated to further understand the impact of both learning methods on inducing measurable short- and long-term competence, and performance gains with focus on board-certified physicians and other healthcare professionals.

List Of Abbreviations

CME: Continuing Medical Education; HCP: Healthcare Professional; PPCP: PinPoint Case Platform; QS-PCJ: QuickScan plus Personalised Case Journey(s); UEMS-EACCME: European Union of Medical Specialists - European Accreditation Council for Continuing Medical Education

Availability Of Data And Materials

The data that support the findings of this study are available from Ismar Healthcare ([email protected]) but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are, however, available from the authors upon reasonable request and with permission of Ismar Healthcare.

Ethics Approval And Consent To Participate

Because the study is not to be considered as an experiment with human beings and not related to any medical intervention, formal institutional review was not considered necessary. In addition, participants were recruited via invitation of their supervisors in various universities in different countries, but participation was in a personal capacity, and not institution-related. Nevertheless, the study complies with Good Research Practices, following the principles outlined in the Declaration of Helsinki through obtaining written informed consent from all participants before the start of this non-medical study. All data was handled, processed and stored in line with GDPR, ensuring that only personal data relevant to the study (country, year of residency) were collected.

Acknowledgments

The authors gratefully acknowledge the efforts of all experts, residents and Ismar Healthcare staff members who contributed to the development and maintenance of the PinPoint Case Platform.

Disclosure Statement

AB has received meeting support and honorarium from Sanofi, Astellas, Janssen, Bayer and AAA Pharmaceutical. NRMM, JS, JT, FVdA and TZ declare no competing interests related to the present study. NH and JY are employees of Ismar Healthcare. LS, LVR and HS are partners in Ismar Healthcare and e-HIMS. MJS has received honoraria from Astellas for lectures/chairmanship. BFT has received grants/research supports or honoraria from Amgen, Astellas, Janssens, Ferring, Sanofi, Bayer and Myovant. MCM has received speaker and/or consultancy honorary from Apogepha, Astellas, Dr. Willmar Schwabe, GSK and Sanofi-Aventis.

Supplemental data

Supplemental data for this article can be accessed online at https://doi.org/10.1080/21614083.2022.2153438.

Additional information

Funding

The research and development of the individualised learning platform was funded by a grant received by the Flanders Agency for Innovation and Entrepreneurship (VLAIO) (grant agreement HBC.2019.2822).