Online CME: an effective alternative to face-to-face delivery

Abstract

Background: The Pharmacotherapies Accreditation Course (PAC) is a continuing medical education (CME) course designed to prepare practitioners for accreditation as pharmacotherapies prescribers for opioid dependence. The course incorporates a preparation stage, a workshop stage and a clinical placement component. The PAC continues to be successfully delivered in face-to-face mode since 2001. From 2003 onwards, an online alternative of the PAC was also implemented.

Aims: The aim of this study was to evaluate the effectiveness of an online alternative to an existing face-to-face CME workshop in preparing practitioners for accreditation as a pharmacotherapies prescriber for opioid dependence.

Methods: Participants were 62 practitioners who undertook the PAC between 2003 and 2006. A pretest/posttest-control group design was used, with outcome measures across the domains of knowledge, skill, and attitudes, together with a course feedback survey for both the online and face-to-face modes of the course.

Results: Results demonstrate that the online CME mode was equally as effective as the face-to face mode in preparing participants for their role in the treatment and management of opioid dependence, and was also rated highly by participants.

Conclusions: The findings have implications for the effective design and delivery of e-learning environments for professional practice, in terms of equipping participants with requisite clinical knowledge and skills and facilitating the development of attitudes congruent with professional practice.

Introduction

Clinicians are encouraged to regularly update their knowledge and skills, and maintain the standards of clinical excellence required for the optimal care of patients through CME that is aimed at improving performance and optimising the outcomes of practice (Fox & Bennett 1998; Grant & Stanton 2003). Rapidly evolving technologies have made possible the development and implementation of high quality web-based CME courses that can meet the demands of a diverse community of practitioners for convenient and flexible access.

Situated Learning Theory (SLT) provides a particularly relevant framework for online CME. Central tenets of this theory emphasise ‘authentic’ learning within a ‘community of practice’, facilitated by an expert practitioner and supported by interaction and collaborative knowledge construction. The term ‘cognitive apprenticeship’ is often used to emphasise the importance of activity in knowledge construction, and to highlight the ‘situatedness’ of learning in a particular context (Brown et al. 1989; Lave & Wenger 1991; Young 1993; Barab & Duffy 1998; Stein 1998). Key components of situated learning include the use of realistic and problem-centred learning activities to promote learner motivation and engagement, access to expert guidance and feedback, ‘scaffolding’ that provides a cognitive framework for learning, and opportunities to consider multiple perspectives through interactive discussion and reflection (Brown et al. 1989; Lave & Wenger 1991; Young 1993; Stein 1998).

Table

Practice points
•Online CME can be as effective as face-to-face teaching in the development of knowledge, skills, and attitudes for professional practice.
•Situated Learning Theory (SLT) provides a relevant framework for the design of online CME.
•A CME program that is based on sound educational principles can result in quality learning experiences and outcomes, regardless of mode of presentation.

Online CME environments that provide learners with flexible opportunities to engage in authentic, interactive, and self-directed learning activities, have been shown to not only facilitate participant engagement and collaboration (Francis et al. 2000; Zimitat 2001; Harris et al. 2002; Liaw et al. 2002), but also to improve learners’ acquisition of knowledge and skill and facilitate transfer beyond the initial context of learning (Davis et al. 1995; Davis et al 1999; Zimitat 2001; Mazmanian & Davis 2002; Curran & Fleet 2005).

However, most research studies that have evaluated the efficacy of web-based CME lack methodological rigour, predominantly relying on participant satisfaction data as an outcome measure or using a single-arm pretest posttest design with no control group (Francis 2000; Kronz et al. 2000; Harris et al. 2001; 1999). Only a small number of studies have made use of a pretest posttest control group design (Curran et al. 2000; Hinkka et al. 2002) or a randomised controlled trial (Fordis 2005; Harris et al. 2002). Even with these controlled trials, it is difficult to judge whether demonstrated improvement in clinician knowledge and/or skills is a result of mode of delivery or exposure to educational content (Fordis 2005), as the outcomes of an intervention group receiving online CME are often compared to those of a control group receiving no CME (Curran et al 2000; Harris et al. 2002; Hinnka et al. 2002).

To be more confident that web-based CME is indeed an effective alternative to face-to-face learning, it is imperative to address the methodological shortcomings of previous research and systematically appraise the potential of web-based CME to equip practitioners with the requisite clinical knowledge and skills necessary for ongoing professional practice. This paper reports the results of one such Australian study involving a State Government CME initiative – the Pharmacotherapies Accreditation Course (PAC).

The pharmacotherapies accreditation course

The Pharmacotherapies Accreditation Course is designed by New South Wales Health and the Centre for Innovation in Professional Health Education and Research (CIPHER) at the University of Sydney to accredit practitioners to prescribe pharmacotherapies for opioid-dependence. It focuses on developing six aspects of clinical competence: attitudes and professional practice, patient assessment, development of a treatment plan, management of co-morbidity, patient management, and quality improvement. Course content is built around a series of topical cases that focus on treatment approaches to opioids and heroin addiction and on maintenance treatment with methadone, buprenorphine and naltrexone. Each case has a trigger statement and a series of challenge questions. An example is provided in Appendix 1.

PAC design

Figure 1 provides an overview of the course, showing alternative face-to-face and online modes of delivery and the features that are common to both modes.

Figure 1. PAC Overview.

Both modes involve (1) a preparation stage consisting of a training manual and online cases; (2) a workshop stage culminating in a NSW Health accreditation examination; and (3) a subsequent half-day clinical placement with an experienced pharmacotherapies prescriber (essential to complete the NSW accreditation process). The focus of the study reported here is the workshop component of the course.

The PAC has been successfully delivered in face-to-face mode since 2001 (for example 115 participants undertook the course between 2001 and 2003), with consistently high participant pass rates and similar high levels of participant satisfaction ratings. An alternative online mode of the PAC was developed, tested, and implemented concurrently from 2003 onwards with the aim of increasing program flexibility and to address access issues for practitioners.

Methods

Aim

The aim of this study was to evaluate the effectiveness of an online alternative to an existing face-to-face CME workshop in preparing practitioners for accreditation as a pharmacotherapies prescriber for opioid dependence.

PAC workshop design

The design of both the face-to-face and online workshop modes was based on the SLT framework outlined earlier, together with our considerable experience in the design of online learning environments (cf. Ryan et al. 2004), and on other published delivery frameworks for web-based learning environments (Young 1993; Herrington & Oliver 2000; Oliver 2000; Bennett et al. 2001; Reeves et al. 2002; Segrave & Holt 2003).

Both workshops incorporate facilitated small-group case-based learning and use the same clinical cases. They are equivalent in workload and resulting CME points, and participants are required to pass the same NSW Health accreditation examination that is held at the end of the workshop to test knowledge of the medical management of opioid dependence. Figure 2 provides details of the structure of the online workshop.

Figure 2. PAC Online workshop structure.

Study design

The current investigation addresses methodological constraints of previous CME studies by (1) employing a pretest/posttest-control group design (the well-established face-to-face mode was used as the non-randomised control); (2) assessing knowledge, skill and attitude outcomes; and (3) evaluating the nature and extent of change reflected in each of these outcomes in both online and face-to-face modes of the program. This study was approved by the University of Sydney Human Research Ethics Committee. Figure 3 shows the overall study design.

Figure 3. Study design.

All participants completed a multiple choice question (MCQ) pretest prior to attending the workshop and a MCQ posttest immediately after the workshop. Two equivalent sets of MCQs were developed specifically for the purposes of the study – set 1 test and set 2 test, with participants completing Set 1 as a pretest and Set 2 as a posttest. The same content expert (author JB) who developed the NSW Health accreditation exam constructed the MCQ tests, which were benchmarked against the accreditation examination in terms of content and difficulty, with no duplicate questions. Each test comprised three types of MCQs; single best answer, multiple true false; and extended matching items.

In addition, all participants completed three questionnaires prior to and at the end of the workshop. These included two self report measures of skills developed by author JB for the purposes of this study: (1) a self-evaluation of skills in assessing dependency problems in areas such as problem identification, history taking, patient examination, diagnosis, providing feedback, and information on treatment options and management; and (2) a self-evaluation of skills in treatment planning, risk assessment, and ongoing management. The third questionnaire was the previously validated 14-item Abstinence Orientation Scale (Caplehorn et al. 1998) designed to measure attitude and orientation to harm minimisation. Participants also completed a course feedback survey on workshop completion that required them to rate how adequately the workshop had met their professional preparation needs for (1) diagnosis and management of opioid dependent patients; (2) changing perceptions about the needs and behaviours of opioid dependent patients; and (3) improving confidence to practise as a pharmacotherapies prescriber.

Participants

All practitioners who enrolled in a PAC course between 2003 and 2006 were invited to participate in the study. The extended data collection period was due to the small number of enrolments in each PAC and the need to gather sufficient data for statistical analysis. Participants self-selected into either the face-to-face or online mode of the program based on their preference of learning mode or geographic location in New South Wales-random allocation was not possible.

Analysis

Scores on the knowledge tests, skills tests, and attitudes scale were analysed using the Wilcoxon Signed Ranks test for paired samples and the Mann-Whitney test for two independent samples. The Wilcoxon Signed Ranks test enabled a within-group comparison of pre- and posttest scores for the face-to-face and online participant groups, while the Mann–Whitney test permitted a between-group comparison along the same outcome measures.

Results

Of 87 practitioners who undertook a PAC course between 2003 and 2006, both pre- and post evaluations of knowledge, skills and attitudes were completed by 62 practitioners: 35 participants in the face-to-face mode and 27 in the online mode. This represents a 71% response rate. The post workshop satisfaction survey was completed by a total of 71 participants representing an 81% response rate.

The majority of participants were general practitioners (GPs) and registrars, located in a range of metropolitan and regional practices across the state of New South Wales. Gender data was available for 56 participants – 30 male and 26 female.

Table 1 outlines results of Wilcoxon Signed Ranks paired sample analysis of participant pre- and post scores on the knowledge test, self-report measures of skills, and the attitudes scale.

Table 1.  Paired samples analysis

	n	Pre-workshop mean (SD)	Post workshop mean (SD)	p
Knowledge test
all participants	62	52.8 (13.2)	67.1 (12.2)	<0.001
face-to-face	35	52.5 (13.4)	68.3 (12.0)	<0.001
on-line	27	53.2 (13.3)	65.5 (12.4)	<0.001
Skills in assessment
all participants	58	34.4 (7.8)	41.8 (6.6)	<0.001
face-to-face	31	35.3 (6.3)	41.8 (6.0)	<0.001
on-line	27	33.3 (9.2)	41.8 (7.3)	<0.001
Skills in treatment planning
all participants	57	14.6 (5.5)	22.7 (3.8)	<0.001
face-to-face	31	14.6 (5.3)	23.1 (3.6)	<0.001
on-line	26	14.6 (5.9)	22.3 (4.1)	<0.001
Attitudes
all participants	56	35.8 (8.1)	29.5 (6.8)	<0.001
face-to-face	29	36.7 (8.3)	28.1 (6.8)	<0.001
on-line	27	34.8 (8.0)	31.1 (6.5)	<0.01

Table 2 reports results of the Mann–Whitney independent sample analysis.

Table 2.  Independent samples analysis

	n	Mann–Whitney U	p
Knowledge test
face-to-face	35	451	=0.760
on-line	27	421.5	=0.469
Skills in assessment
face-to-face	31	393	=0.690
on-line	27	410.5	=0.901
Skills in treatment planning
face-to-face	31	392	=0.860
on-line	26	371.5	=0.608
Attitudes
face-to-face	29	341	=0.407
on-line	27	298	=0.124

Knowledge

The pre- and post knowledge tests were each scored out of a maximum mark of 100. Overall, scores on both the pre- and posttests were uniformly high for participants in both the face-to-face and online workshops. Significant differences were found between the pretest and posttest means within groups. Comparison of the pretest and posttest means between groups (face-to-face and online) yielded no significant differences.

Skills

1. Self-evaluation of skills in assessment of dependency problems

The pre- and post evaluations were each scored out of a maximum mark of 55. Significant differences were found between pretest and posttest means of all participants. Comparison of the pretest and posttest means between groups showed no significant differences.

1. Self-evaluation of skills in treatment planning

The pre- and posttest were each scored out of a maximum mark of 30. Significant differences were demonstrated between pretest and posttest means for all workshop participants. Comparison of the pretest and posttest means between groups yielded no significant differences.

Attitudes

A high score on the Abstinence Orientation Scale (Caplehorn et al. 1996) which is marked out 70, indicates support for the goal of abstinence and limitation of maintenance treatment whilst a low score indicates an orientation to harm minimisation and support for methadone maintenance. Significant differences were demonstrated for all participants between pretest and posttest.

A between-group comparison shows that the online participants had more favourable attitudes to methadone maintenance at pretest than the face-to-face participant group but these differences were not significant. At posttest, face-to-face participants had more favourable attitudes than online participants but these differences were also not significant.

Participant satisfaction

The course feedback survey used a 5-point Likert scale that ranged from ‘less than adequately’ to ‘more than adequately’. Although participants self-selected into their choice of learning mode, both face-to-face and online workshops were consistently rated highly by participants. Satisfaction outcome means ranged between 3.9 to 4.2 (out of 5) for the face-to-face workshops and 3.6 to 4.3 (out of 5) for the online workshops.

Discussion

The aim of this study was to evaluate the effectiveness of an online alternative to an existing face-to-face CME workshop, with both workshops being preceded by an identical preparation phase. We have addressed the methodological shortcomings of previous research by employing a pretest/posttest non-randomised control group design in which practice effects were controlled by using different but equivalent forms of the knowledge test at pre- and posttest.

Overall, there were significant differences between pre- and posttest means for both groups on the knowledge test, and self-report measures of skills, and the attitudes scale. This indicates that both PAC workshop modes were effective in improving knowledge outcomes and fostering a perception of improved skills and favourable attitudes towards methadone maintenance programs and orientation to harm minimisation. The non significant differences between groups for pre- and posttest means across all outcome measures indicates that both workshop modes were equally effective in improving outcomes.

Interestingly, the between group comparison on the Abstinence Orientation Scale scores showed that participants in the face-to-face workshop had made more of a shift in their attitudes between pre- and post test than online participants. Although not significant, this greater shift in face-to-face participants’ attitudes between pre- and post test may be indicative of the capacity of face-to-face discussion to facilitate greater attitude change. This aspect requires further investigation.

Based on the SLT and other education delivery frameworks and our experience in the design of online learning environments, the PAC workshop was pedagogically designed to build upon learning achieved in the preparation phase. Thus, it is reasonable to conclude that the demonstrated improvements in outcomes at posttest for both modes can be attributed to the combined learning occurring in the preparation and workshop stages of the PAC.

Conclusion

Our results demonstrate that the online mode of a CME course was equally as effective as the face-to face mode in preparing participants for their role in the treatment and management of opioid dependence, and was also rated highly by participants. Addressing methodological shortcomings of previous studies that have evaluated the efficacy of online CME has added strength to our findings.

We have shown that a well designed continuing education course that is based on sound educational principles can result in quality learning experiences and outcomes. Our findings lend support to the view that whilst learners will continue to select the educational mode that best suits their learning needs, it is the quality of the design and delivery that is the key to effective learning, regardless of the mode of presentation.

Future research in this area should also strive for rigorous method, and explore the transfer of knowledge, skill, and attitude gain into the practice setting, identifying factors that facilitate or impede this transfer.

Additional information

Notes on contributors

Greg Ryan

GREG RYAN, PhD, is an Associate Professor in the Faculty of Pharmacy and Director of the Pharmacy Education Unit, University of Sydney, NSW, Australia.

Patricia Lyon

PATRICIA LYON, PhD, is a Senior Lecturer in Medical Education, in the Centre for Innovation in Professional Health Education and Research (CIPHER), Faculty of Medicine, University of Sydney, NSW, Australia. She also coordinates the Master in Medical Education program in the Faculty of Medicine.

Koshila Kumar

KOSHILA KUMAR, MA, is a Research Officer and PhD candidate in the Centre for Innovation in Professional Health Education and Research (CIPHER), Faculty of Medicine, University of Sydney, NSW, Australia.

James Bell

JAMES BELL, FRACP, MD, is Director of The Langton Centre, NSW, and is also Associate Professor (Conjoint) in the Department of Medicine, University of New South Wales, NSW, Australia.

Stewart Barnet

STEWART BARNET, BA, Dip Teach, Grad Dip Ed.Tech, is the Manager of Educational Design, in the Centre for Innovation in Professional Health Education and Research (CIPHER), Faculty of Medicine, University of Sydney, NWS, Australia.

Tim Shaw

TIM SHAW, PhD, is an Associate Professor of Health Workforce Education and Director of program development in the Centre for Innovation in Professional Health Education and Research (CIPHER), Faculty of Medicine, University of Sydney, NSW, Australia.

Appendix 1

Lydia is a 22 year old injecting opioid user who commenced heroin use some four years ago. She presents to you because she thinks she is pregnant as she has not menstruated for the past three months. On examination you see a thin pale woman who weighs 45 kg. There is evidence of track marks in both antecubital fossae. She has a soft abdomen with no palpable masses. The remainder of the examination is normal.

Challenge questions

• What is your differential diagnosis for Lydia's failure to menstruate? List up to 3 diagnoses.

• A pregnancy test comes back positive. What are your next steps?

• Lydia has decided she wants to keep the baby. You have established that she is opioid dependent. What advice do you give about what she should do?

• The patient indicates that her preferred treatment would be to commence methadone and then to withdraw as quickly as possible, to be drug free by the time of delivery. When is the optimal time to withdraw a pregnant, opioid-dependent woman?