Does an outcome-based approach to continuing medical education improve physicians’ competences in rational prescribing?

Abstract

Background: Continuing medical education (CME) is compulsory in Iran, and traditionally it is lecture-based, which is mostly not successful. Outcome-based education has been proposed for CME programs.

Aim: To evaluate the effectiveness of an outcome-based educational intervention with a new approach based on outcomes and aligned teaching methods, on knowledge and skills of general physicians (GPs) working in primary care compared with a concurrent CME program in the field of “Rational prescribing”.

Method: The method used was cluster randomized controlled design. All GPs working in six cities in one province in Iran were invited to participate. The cities were matched and randomly divided into an intervention arm for education on rational prescribing with an outcome-based approach, and a control arm for a traditional program on the same topic. Knowledge and skills were assessed using a pre- and post-test, including case scenarios.

Results: In total, 112 GPs participated. There were significant improvements in knowledge and prescribing skills after the training in the intervention arm as well as in comparison with the changes in the control arm. The overall intervention effect was 26 percentage units.

Conclusion: The introduction of an outcome-based approach in CME appears to be effective when creating programs to improve GPs’ knowledge and skills.

Introduction

It has, since more than a decade, been a challenge to provide effective and relevant programs for continuing medical education (CME) that support physicians in their efforts to stay current with new developments in medicine. Outreach visits, reminders, and audits with targeted feedback has been shown to have some effect (Davis et al. 1995; Cantillon & Jones 1999; Jamtvedt et al. 2006). Since making CME compulsory for all physicians and other health professionals in 1991, concerned authorities in Iran, including the educational centres at some medical universities, have been working to meet the challenge of developing effective approaches. Traditional, mainly teacher-centred, didactic CME programs are usually not successful in changing doctors’ performance, but interactive workshops may be successful (Oxman et al. 1995; Thomson O’Brien et al. 2001), in particular, if participants’ activity is promoted (Davis et al. 1999). However, further development of program structure and implementation is needed in Iran as a gap has been shown between established needs and the proffered programs (Shirazi et al. 2004).

Table

Practice points

There is a need to improve the rational prescribing behavior of physicians and the CME in this field in Iran. OBE approaches have been successfully used in undergraduate medical education and is starting to find its way into CME. By using an outcome-based approach in an educational intervention about rational prescribing, a significant improvement in the competencies of physicians was achieved. The improvement took place although the teachers were relatively new to the OBE approach. It is important to evaluate how educational interventions work and impact on behavior in the “real world”.

A new approach to teaching and learning is outcome-based education (OBE), which was initially proposed for undergraduate training (Harden 1999), but which has been increasingly used in CME as well (Harden 2002; Moore & Tonniges 2004; Harden 2006; Harrison & Mitchell 2006). OBE can be seen as an overarching approach that can influence the entire process of education, mediating decisions about the content, formulation of aims, educational strategies, teaching methods, assessment procedures, and the educational environment (Harden 2007). Such an approach could be deemed appropriate to mitigate the challenges for improving CME for medical doctors in Iran.

We chose training in “Rational prescribing” as a topic for the CME intervention, as prescribing of drugs is one of the most important therapeutic tools at the disposal of a physician. Furthermore, there are considerable differences in the choice and quantity of drugs prescribed for the same ailment as well as in the quality of the prescriptions themselves (Watkins et al. 2003; Muijrers et al. 2004). There are also convincing data that show an excessive prescribing behavior in Iran (World Health Organization 2003). Some studies point to inappropriate, even irrational drug prescribing (Moghadamnia et al. 2002; Cheraghali et al. 2004; Gholamreza & Meimandi 2005). Training in rational prescribing has been part of the CME programs in Iran for some time, but the impact has not been evaluated.

Therefore, our general research question was that whether an outcome-based approach could be used in the CME context in Iran to improve doctors’ prescribing of medicines. In an initial study (Esmaily et al. 2008), we used a modified Delphi process to identify outcomes and develop a curriculum for a CME course in rational prescribing for general physicians (GPs) working in primary care. Content and core curricula were designed for six topics: (1) Principles of prescription writing, (2) Adverse reactions to drugs, (3) Drug interactions, (4) Injections, (5) Antibiotic therapy, and (6) Therapy with anti-inflammatory agents, including corticosteroids. The content of each topic was validated by a team of experts in the field. Our more specific research question became whether a CME program for GPs, based on the developed outcomes and using appropriate teaching methods, could improve their knowledge and skills in prescribing of medicines.

The aim of this study was, therefore, to assess the effects on knowledge and skills of GPs participating in a CME program on rational prescribing with an outcome-based approach and to compare these with participants in a concurrent CME program.

Methods

The study was set up with a cluster randomized control design, where individual doctors were the unit of analysis.

Study location

Study participants were recruited from GPs practising in primary care in six cities in the East Azerbaijan province in Iran, not including the capital city of Tabriz. East Azerbaijan is one of 30 provinces in Iran (26 provinces at the time of the study) and has a total population of 3.5 million. Three cities in the Northern part of the province (in total 217,000 inhabitants) were matched with three cities in the South (in total 265,000 inhabitants), based on a ranking compiled in 2003, which looked at human development factors (economic status, health services, education, sports facilities, agriculture, and communication facilities) (Hekmati Farid 2003). This geographic separation was used to diminish the risk of participants interacting with each other and thereby potentially introducing confounding factors. The group of northern cities was then randomly selected as the intervention group.

Participant selection

Of the population of 185 GPs in the selected cities, 159 were identified as having a contract with one or more of the three biggest social insurance organizations in Iran. Those GPs without a contract were either working in a specialized clinic or working part-time with only a few patients. More than 85% of the Iranian population use one of these insurance organizations to pay their drug expenses (World Health Organization 2003). These organizations keep records of all reimbursed prescriptions, which allows for following the prescribing behavior of individual doctors. GPs in Iran do not have a formal specialization and work mostly in single, private practices in the communities, or, to a lesser extent, in public health centres. As for all medical professionals, it is compulsory for GPs to take part in CME and amass a specified amount of credits per 5-year period.

The 159 GPs received written invitations aux mains to participate in the CME program on rational prescribing from a representative of the Educational Development Centre (EDC) of Tabriz University of Medical Sciences. The letter included information about the program and emphasized an incentive in the form of CME credit points free of charge, which they would receive upon completion of the course. All 159 responded by returning a confirmation form to indicate whether or not they were interested in participating.

Trainer selection and development of the course plan

The 17 trainers in the intervention and control programs were all faculty members of Tabriz University of Medical Sciences and experienced CME trainers. All had participated in previous workshops on educational methods. They were chosen among those who ranked highest on the yearly evaluation of faculty performance and who had already been involved in teaching in programs on rational prescribing. To maintain internal validity, trainers were matched in pairs, based on their evaluation results, and the topics they had taught previously. From this list of pairs, one group of eight trainers (three women, five men; four medical specialists, four pharmacists) was assigned to teach in the control group in the conventional manner, and the other group of eight trainers (two women, six men; four medical specialists, four pharmacists) was assigned to the intervention program. One trainer (a pharmacist) was later added by the curriculum development group to complement the medical specialist during the injection topic of the intervention program.

The training program for the teachers in the intervention group was conducted during a 3-day workshop, 1 month prior to the CME intervention. The workshop contained sessions on OBE, adult learning, communication skills, training methods, and how to design training sessions. Those responsible for the workshop were all faculty members working in the medical education field. The training began with a 1-day introduction to OBE, followed by discussions about the previously determined outcome indicators for rational prescribing. The common course plan as well as lesson plans for each topic (including duration and educational methodology), were finalized by the eight selected teachers themselves, based on the learning outcomes. An example of how the lesson plan was developed is shown in Box 1.

The educational programs

Each educational program was offered on two occasions at the EDC in Tabriz, in August 2006, for the control group and in September the same year for the intervention group. The number of participants in each course was thus about half of the total number of GPs in each study arm (28 and 30 respectively in the intervention groups; 29 and 25 respectively in the control groups). The CME program for the control group was based on the already existing program on the same topic and was run following the current traditional, mainly didactic approach, i.e., teachers trained participants in the same way as in previously conducted programs, which were lecture-based. In the intervention group, the training was based on the lesson plans from the teacher training workshop, using interactive and learner-activating teaching methods, e.g., activating presentations, question/answer, case studies, case reports, large and small group discussions, and role playing. Supplementary self-learning educational materials were sent to the intervention group after completion of the program.

The programs were offered on different occasions to minimize interactions between participants. The need for full attendance for the duration was emphasized in both programs. The physical environment was the same for both groups and the number of participants in both programs were almost identical. While both programs were offered over two days, the number of hours differed to reflect the different educational methods used, 11 h for the control group using didactic techniques and 16 h for the intervention group, which used more interactive learning methods.

Evaluation tools

Knowledge and skills in rational prescribing were assessed in both groups at the start of the program and after 1 month. The test was designed by a group of CME expert trainers in rational prescribing and validated by a team of experts in the field (Esmaily et al. 2008). Reliability of the test after a pilot study among 29 GPs, not participating in the study, was determined to be 0.74 (Cronbach's coefficient alpha). The test consisted of 30 questions with a maximum score of 53. Participants’ knowledge was assessed with multiple choice and short answer questions and their prescribing skills were tested with two case scenarios and three copies of actual “irrational” prescriptions.

In order to better assess how well the intervention group followed the OBE approach as well as to provide a baseline with the control group, observations were made with the help of a checklist developed at the EDC. Two observers, H. M. Esmaily and a GP (both of whom were planners and educators of teacher training programs at the EDC), assessed the trainers’ and participants’ activities in both the large and small group activities of the intervention groups. In the control groups, the same two observers rated only the activities of the trainers, as small group learning activities were not used in this group. All six topics of the educational programs were assessed using a 3-point Likert scale regarding level of accomplishment (1 = Fully, 2 = To some extent, 3 = Not at all).

Analysis

The intervention and control groups were compared with respect to age and work experience using independent samples t-test. For gender, a χ² test was used. Test scores were compared as repeated measurements pre- and post-intervention. No personal identifier was included in the data set and tests for independent observations were therefore used.

Differences in test scores were first computed pre- and post-test and compared between the intervention and the control groups using the Mann–Whitney U test. The intervention effect was analyzed as the interaction term of being in intervention group at post-intervention, in an ordinary least squares regression model. The interaction term corresponds to the effect of the intervention adjusted for the development over time in the control group. A p-value of <0.05 was considered as significant. All quantitative data were analyzed using the statistical program SPSS 15 (www.spss.com).

Ethical approval for the study was received from the Ethical Vetting Committee of the Iranian Ministry of Health and Medical Education.

Results

Based on the north–south random division of cities, 74 physicians were placed in the intervention group and 85 in the control group. Although 66 of the invited GPs in the intervention program and 71 in the control group had confirmed participation, only 58 (88%) and 54 (76%) actually attended the programs, respectively (Figure 1). The characteristics of the 47 GPs, who declined participation or did not attend following acceptance, were similar to those of the participants (Table 1).

Figure 1. Flowchart of participation in the study.

There was neither significant difference in terms of sex, age, or work experience (Table 1), nor in knowledge or skills (Table 2) between the control and intervention arms prior to participation in the educational programs.

Table 1.  General characteristics of eligible GPs^a for participation in the educational programs

General characteristics	Non- participants N = 46 ^a	Participants N = 112	p-value	Intervention N = 58	Control N = 54^b	p-value
Male n (%)	37 (80)	77 (71)	0.24	42 (72)	35 (70)	0.78
Age (in years) mean (SD)	42 (9.2)	39.9 (6.5)	0.07	39.2 (6.8)	40.7 (6.1)	0.23
Work experience mean (SD)	11.5 (7.2)	10.7 (4.9)	0.43	10.3 (5.2)	11.1 (4.4)	0.42

Note: ^a One 82 years old GP with 53 years work experience excluded from non-participant group before analysis.

^b Four missing cases for sex.

After the program, the intervention group demonstrated a significant improvement, both in comparison with the pre-test as well as in comparison with the control group (Table 2). No such improvements were seen in the control group. The intervention effect on the total score of the outcome-based intervention program was 26 percentage units.

Table 2.  Total score (maximum = 53) for the knowledge and skills test for GPs in the intervention and control groups

	Pre-test			Post-test			Intervention effect
	Intervention mean (SD) (n = 58)	Control mean (SD) (n = 54)	p-value	Intervention mean (SD) (n = 58)	Control mean (SD) (n = 53)	p-value	Intervention effect (SE)	p-value
Score	22.2 (4.7)	21.8 (4.6)	0.626	37.3 (3.8)	23.1 (3.8)	<0.001	13.8 (1.1)	<0.001
Percentage (%) of total score	42 (8.9)	41 (8.7)	–	70 (7.2)	44 (7.2)	–	–	–
Diff. Pre-post (%)	–	–	–	29	2.5	–	26	–

By breaking down the scores into the six different topics of the rational prescribing curriculum, we found that a significant improvement was achieved by the intervention group in all six areas, with a range of the intervention effect from 15–39 percentage units (Table 3). However, the number of questions that showed a significant change varied among the topics. All four questions about the principles of prescription writing showed a significant improvement. For the other topics, seven of nine (Adverse reactions to drugs), three of four (Drug interactions), two of three (Injections), three of 11 (Antibiotics), and five of seven (Therapy with anti-inflammatory agents) questions showed a significant change.

Table 3.  Total score for the knowledge and skills test for GPs in the intervention (n = 58) and the control (n = 54) groups for the six different curricular topics

	Pre-test			Post-test			Intervention effect
Topic (Maximal score)	Intervention mean (SD)%*	Control mean (SD)%	p-value	Intervention mean (SD)%	Control mean (SD)%	p-value	Intervention effect (SE) percentage	p-value
1. Principles of prescription writing (19)	7.2 (3.1) 33	6.7 (3.0) 35	0.58	14.4 (2.5) 76	7.4 (2.6) 39	<0.001	6.5 (0.8) 34	<0.001
2. Adverse reactions to drugs (9)	4.2 (1.1) 47	4.2 (1.2) 47	0.90	6.3 (1.1) 70	4.3 (1.2) 48	<0.001	2.0 (0.3) 22	<0.001
3. Drug interactions (4)	2.1 (1.1) 53	2.2 (0.9) 55	0.37	2.8 (1.0) 70	2.3 (0.9) 58	0.007	0.6 (0.3) 15	0.022
4. Injections (3)	1.9 (0.7) 63	2.0 (0.4) 67	0.29	2.5 (0.7) 83	2.0 (0.5) 67	<0.001	0.5 (0.2) 17	0.001
5. Antibiotic therapy (11)	4.0 (2.0) 36	4.0 (1.9) 36	0.91	5.9 (1.7) 54	4.1 (1.4) 37	<0.001	1.8 (0.5) 16	<0.001
6. Anti-inflammatory agents therapy (7)	2.7 (1.2) 39	2.5 (1.2) 36	0.37	5.2 (1.0) 74	2.7 (1.1) 39	<0.001	2.4 (0.3) 34	<0.001

Note: *Percentage of maximum score for each topic.

The observations of the training sessions revealed that the general characteristics of the teaching and learning environment were more favorable for active learning in the intervention as compared to the control groups (Table 4).

Table 4.  Total observational checklist scores for the intervention and control groups from the two observers covering the six different curricular topics

Item	Intervention group			Control group
	Fully accomplished	To some extent	Not at all	Fully accomplished	To some extent	Not at all
1. Educational goals were clearly explained in the beginning	10	2	–	–	2	10
2. Opportunity for active participation in discussions for all participants	12	–	–	–	4	8
3. Trainers encourage participants to actively participate in all activities	11	1	–	–	1	11
4. Trainers manage time well	7	5	–	2	10	–
5. The class has a friendly atmosphere	12	–	–	1	10	1
6. Educational materials are given to participants	10	2	–	–	–	12
7. Educational references related to the topic are presented	7	3	2	–	–	12

Box. 1 Example of how teaching methods and learning activities were developed related to the outcome “Prescribe the appropriate number of injections”

Outline	Teaching method	Learning activities	Teacher's assessment
State objectives of lesson (5 m.)	Presentation
Consideration of real needs for prescribing injections (15 m.)	Questions/answers Activating presentation	Answers/questions Brain storming	Quality of participation
Mechanism of injections (15 m.)	Activating presentation Questions/answers	Questioning Answer to questions	Answers to teachers’ questions
Indications for injections (15 m.)	Activating presentation Questions/answers	Questioning Answer to questions	Answers to teachers’ questions
Important factors in prescribing injections (20 m.)	Show samples of real prescriptions	Group discussion	Results from individual examples
Prevalence of using injections in the world and in Iran (10 m.)	Activating presentation	Bring up and discuss own knowledge and experience	Quality of participation
Conclusion [review of the topic, answer to questions] (10 m.)	Presentation	Feedback from the group	Quality of participation

Group activities in the intervention program were facilitated in a supportive fashion. The observation checklists for the group work showed that the goals for the group work were clearly explained and that the small group members actively participated in the discussions. Participants were able to interact face-to-face with ease and an active listening style dominated.

Discussion

We found an unequivocal overall improvement of 26 percentage units in the knowledge and skills of general physicians after participation in the outcome-based CME program on rational prescribing. Improvement was seen in all six topics of the curriculum.

Compared with traditional didactic lectures, an active learning approach with alignment between relevant topics and the curricular design has been found to be more effective in changing physicians’ behavior (Sohn et al. 2004; Bloom 2005). Even when learners have lower perceptions about the ability of the sessions to meet learning objectives, no detrimental effects have been found from going over to an active-learning methodology (Haidet et al. 2004). The difference between the intervention and control groups in our study could be further related to the use of an outcome-based approach which stresses the importance of constructively aligning (Biggs 2003) the curriculum with outcomes, content, educational methodology, and assessment – all building upon each other. This provides a structure that teachers can use when designing their teaching and learning activities. Based on the observations from the sessions, teachers in the intervention group followed this structure consistently, which could be an additional explanation why a significant improvement was seen in all the topics.

Another explanation for the success of the intervention program could be the effort that was made to relate the curriculum and its content (including the examples that were used) to the real world that the participants worked in. For instance, actual prescriptions, which included examples of drug interactions and irrational prescribing, were used, as well as discussions of real-life professional situations. Throughout all the sessions, a practical rather than a theoretical focus was emphasized. In interpreting the results, it is important to consider issues of feasibility and how the approach described here can impact the future design of CME programs.

As described above, it should be noted that we have evaluated the whole process of encouraging CME teachers to use outcomes as a base for the design of curriculum and lesson plans. Teachers in the control group were similarly encouraged to use the methods they found most suitable to reach the goals set in the traditional program on rational prescribing. This implies that we cannot separate the effects of using outcomes from using more active learning methods as we compare the effects of the training sessions as a whole. However, we maintain that our evaluation is appropriate for assessing how the introduction of an outcome-based approach impacts on the outcome of CME programs in the Iranian setting.

While the number of teachers in each program was similar, teachers in the intervention group taught together in pairs. The benefits of being able to answer questions and conduct discussions together as well as more effectively build upon each other's presentations of the topic may be one of the reasons for the increased effectiveness of the intervention program.

Regarding feasibility, it is important to consider the costs of an outcome-based approach. The initial set-up to determine the outcomes and develop the curriculum required a large amount of time and a large number of people who were involved in the identification of the outcomes. In this study, there were costs involved in turning these outcomes into a program. The increase in costs was primarily due to the fact that such a course did not previously exist. These costs should therefore be seen as a function of starting a new course and there is little reason to assume that they need to be repeated. Moreover, if there is such a difference in outcomes between the control and the intervention groups, the cost may be deferred if the outcome on knowledge and skills also result in an increase in rational prescribing, which is believed to save costs both for individuals and society (Meyer et al. 2001).

The fact that participants came from only one province in Iran could reduce the generalizability of the findings. However, there is no existing evidence that these doctors would behave differently compared to the doctors in similar settings elsewhere in Iran. Moreover, the fact that the characteristics of the control and intervention groups were similar, coupled with the fact that there was no significant difference in mean pre-test scores, suggest that the groups were evenly matched.

A limitation of the study was the drop out of eight participants from the intervention group and 17 from the control group. Stated reasons were sickness, travel, and sudden scheduling conflicts. One interesting observation was that the incentive to encourage participation that we had settled on – offering the program and CME credits free of charge – actually seemed to contribute to the late decision by these would-be participants not to attend as they had not made any advance payment. Another limitation was the choice of not preserving the identification of the doctors in the coding of the tests, which prevented us from following the progress of individual participants.

Those members of the research team, who took part in the workshop with teachers and who participated as observers during the sessions in the intervention groups, may to some extent have influenced the whole process towards a more favourable situation for the intervention arm. However, efforts were made to minimize such an effect. The other observers were not members of the research team and took part in the sessions more or less on the same grounds as the participants.

It could be argued that the difference in results between the two programs is mainly due to an increase in the length of the intervention course by 5 h. The argument would then be that by increasing the control course by 5 h we would achieve the same results. Perhaps, a more plausible reason for the difference is the approach of the trainers in the teaching and learning activities that resulted from the use of OBE. Trainers began by presenting the outcomes. They then developed the context by illustrating the relevance of these outcomes to everyday professional life with questions to the participants as well as actual examples of irrational prescribing. This approach can be seen as raising the value of the topic in the opinion of the participants (value–expectancy theory), perhaps even contributing to the development of an intrinsic motivation (Biggs 2003). The trainers in the intervention group also interacted with the participants to a higher degree, stimulated the posing of questions, and encouraged discussions (Ramsden 2003). Workshops and small group discussions were used so that participants could learn from and stimulate each other.

With this study, we have found that a CME program, with an outcome-based approach utilizing active learning principles, was more effective than a concurrently offered didactic, lecture-based program in supporting general physicians to develop their knowledge and skills. As in any educational intervention, the extent to which participants modify their behavior after the course is of interest. The impact of this intervention on participants’ prescribing behavior will therefore be evaluated in a forthcoming study.

Acknowledgments

We thank all GPs who participated in the educational programs. A special thanks to all trainers, to the Educational Development Centre of Tabriz University of Medical Sciences, especially Dr Reza Ghafari for logistical and partial financial support, and to the teacher-training team, especially Associate Professor Mahasti Alizadeh. Thanks to Professor Ivan Silver, University of Toronto, and Dr Behrouz Poshtiban for valuable comments and supports. Additional financial support was obtained from the National Public Health Management Centre in Tabriz and the Ministry of Health and Medical Education of Iran.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.

Additional information

Notes on contributors

Hamideh M. Esmaily

Hamideh M. Esmaily is a PhD candidate at the Division of Global Health, Public Health Sciences Department, Karolinska Institutet, Sweden.

Carl Savage

Carl Savage is a PhD candidate at the Medical Management Centre, Department of Learning, Informatics, Management and Ethics, Karolinska Institutet, Sweden.

Rezagoli Vahidi

Rezagoli Vahidi, PhD, is Associate Professor in Health and Nutrition Department, Tabriz University of Medical Sciences, Iran.

Abolghasem Amini

Rezagoli Vahidi, PhD, is Associate Professor in Health and Nutrition Department, Tabriz University of Medical Sciences, Iran.

Saeed Dastgiri

Abolghasem Amini, MD, and Saeed Dastgiri, PhD, are Associate Professors in Medicine Department, Tabriz University of Medical Sciences, Iran.

Hakan Hult

Abolghasem Amini, MD, and Saeed Dastgiri, PhD, are Associate Professors in Medicine Department, Tabriz University of Medical Sciences, Iran.

Lars Owe Dahlgren

Hakan Hult and Lars Owe Dahlgren are Professors in educational Sciences at Medical Education Centre, Department of Learning, Informatics, Management and Ethics, Karolinska Institutet, Sweden.

Rolf Wahlstrom

Rolf Wahlstrom, MD, PhD, is Associate Professor and senior lecturer in Health Systems Research Division of Global Health, Public Health Sciences Department, at Karolinska Institutet, Sweden.