The impact on medical practice of commitments to change following CME lectures: A randomized controlled trial

Abstract

Background: Self-reported commitment to change (CTC) could be a potentially valuable method to address the need for continuing medical education (CME) to demonstrate clinical outcomes.

Aim: This study determines: (1) are clinicians who make CTCs more likely to report changes in their medical practices and (2) do these changes persist over time?

Methods: Intervention participants (N = 80) selected up to three commitments from a predefined list following the lecture, while control participants (N = 64) generated up to three commitments at 7 days post-lecture. At 7 and 30 days post-lecture, participants were queried if any practice change occurred as a result of attending the lecture.

Results: About 91% of the intervention group reported practice changes consistent with their commitments at 7 days. Only 32% in the control group reported changes (z = 7.32, p < 0.001). At 30 days, more participants in the intervention group relative to the control group reported change (58% vs. 22%, z = 3.74, p < 0.01). Once a participant from either group made a commitment, there were no differences in reported changes (63% vs. 67%, z = <0.00, p = 0.38).

Conclusion: Integration of CTC is an effective method of reinforcing learning and measuring outcomes.

Introduction

The concept of commitment to change (CTC) is based on the principles of encouraging and enabling participant reflection on personal goals and values. It is intended to instill in the participant an obligation to improve his/her personal practice behavior. CTC is a central feature of adult learning and promotes the development of expertise. Purkis (1982) is credited with being the first to introduce the CTC process within the framework of continuing medical education (CME). Since then, many CME studies have examined the usefulness of CTCs in explaining self-reported behavior change (Mazmanian et al. 1998, 2001; Lockyer et al. 2001; White et al. 2004).

There is some evidence in the literature that spontaneously generated commitments can lead to self-reported change. Mazmanian (1998) explored the relationship between commitment and change following a lecture on cardiovascular risk using a large sample of physicians and observed that 72% of those making commitments reported making changes. This was further supported by Mazmanian et al. (2001) where those expressing a higher level of commitment were more apt to change than those of a lower level of commitment. Lockyer et al. (2001) reported the percentage of commitments implemented in practice following a course on managing sexual dysfunction delivered in multiple centers. Despite having no control group, Lockyer et al. demonstrated a “dose response” relationship between the time devoted to the course and the number of changes reported. As predicted, the longer the course, the greater the number of reported changes. Because of the design, Lockyer et al. could not, however, report a relationship between making commitments and ultimately changing practice behaviors.

White et al. (2004) demonstrated that CTCs were helpful to presenters in determining congruence of program objectives with intended changes of participants. The study by White and colleagues was not designed to investigate a relationship between CTCs and behavior change but does suggest the value of CTC to presenters. Along with providing meaningful feedback for evaluating similarities among intended changes and program objectives, presenters were able to document unexpected learning outcomes and to enable and reinforce intended behavior change (White et al. 2004).

Previously published randomized, controlled trials have been challenged with issues of sample size. For example, Pereles et al. (1997) utilized a randomized control design to measure the impact of making commitments on behavior change in a course for physicians on healthcare for the elderly. However, only 17 physicians participated in the study (eight in the CTC group and nine in the “non-commitment group”) and only 4 physicians in the non-commitment group responded to the survey at 1-month follow-up. Because the sample size was so small, the authors were not able to demonstrate a significant difference between the control and intervention groups in the impact of commitments on practice behavior. However, they did observe interesting trends in the predicted direction (greater CTC = greater practice change). Likewise, Wakefield et al. (2003) addressed in their discussion the influence of their sample size as a limitation.

The relationship between CTC and self-reported behaviors is critically important because the latter can be significantly related to actual behavior change. Wakefield et al. (2003) found that following participation in a CME program using interactive small groups, physicians who made their own CTCs related to prescribing practices were more likely to implement those changes in actual observed prescribing behaviors. They conclude that “self reported change of prescribing can be a proxy for actual change” (Wakefield et al. 2003, p. 88). Their study further suggests that this relationship between CTC and actual behavior is heavily influenced by the nature of the educational intervention underlying the CTC. Additionally, the influences of how health care commitments are individually indentified continue to be examined. Overton and Mac Vicar (2008) found that two conceptualizations of commitments exist in the literature, behavioral and attitudinal. Furthermore, Overton and Mac Vicar (2008) explore and postulate how antecedents of these two conceptualizations influence choice of commitment. An example described within this article concerns antecedent knowledge of a post-intervention inquiry resulting in a 10-fold increase in behavior change over a group without the antecedent knowledge of a follow-up inquiry (Lewin 1966).

Large-audience CME lecture (as opposed to small group discussion or self-learning using a resource like a journal, book, or video) is a common method of post-residency CME. This method, which typically involves a speaker delivering a lecture to a passive audience, and accepting questions briefly at the end of the lecture, allows for many providers to be exposed to content, and when done over a series of days, permits a large amount of diverse material to be dispersed. Yet, good data exist demonstrating large-audience CME does not, alone, change behavior. Forsetlund et al. (2009) in a Cochrane systematic review found that coupling educational lectures with other interventions like using CTC can “improve professional practice and health outcomes for the patients.”

At a time when CME is searching for meaningful outcomes to demonstrate the efficacy of educational interventions, self-reported CTC appears to be a potentially valuable construct. It is a potential method of reinforcing learning and measuring positive educational outcomes. This study attempts to answer the following questions:

1. Are clinicians who make commitments to change based on choice among a predefined list more likely to report changes in their medical practices than those who are not presented with the list?

2. Are these changes at least as durable as those made spontaneously by a control group? If so, this would present an efficient and highly replicable intervention to enhance CME outcomes. Currently, the literature does not consistently demonstrate the superiority of predefined lecture-derived commitments over those made spontaneously.

Table

Practice points

CTC can be successfully applied to large audience CME and result in positive change in clinical practice. Electronic tools can assist in making CTC theory an “easy to integrate” educational tool. Making commitments, whether selecting them from a predefined list or generating them spontaneously, is positively associated with practice change. Once providers complete a commit to a change cycle, the change persists and remains stable.

Method

Setting and recruitment

During the fall of 2009, email invitations to participate in a CME study were sent to all clinicians (4745) who registered for the Harvard Medical School sponsored CME meeting “Current Clinical Issues in Primary Care” (aka Pri-Med) Conference in Boston. This study focused on one lecture covering common ambulatory psychiatric issues lasting 45 min in a course that offered 16 h of CME credit over 3 days. Those who responded with interest were sent an introductory packet explaining the basic premise of the study. On the day of the lecture, attendees were again invited to participate in the study by enrolling at the back of the auditorium prior to the lecture; this group was randomized by placing every other enrollee in the intervention group.

No monetary incentive was offered for participation in this study, but a discount coupon was provided from a medical book publisher to all participants as a form of thanks following the intervention. Because this was a voluntary recruitment, the participants in this study were randomly selected. No attempt was made to specifically sample attendees based upon degree (e.g., MD, DO, NP, and PA) or upon demographic (e.g., location of practice, age/years in practice).

In the invitation, participants were told that if they agreed to participate, they would be assigned to either the intervention or the control group and that those in the intervention group would be asked to stay for 5 min following the lecture. Furthermore, they were told that being part of the study meant that they would be contacted via email several times over the next few months and asked to answer questions pertaining to the study which would take less than 5 min of their time. Participants were instructed to go to the back of the lecture hall following the lecture to pick up their study packets.

Immediately following the lecture, those participants in the intervention group were instructed to remain after the lecture and to select up to three commitments from a predefined list based on the lecturer's objectives using an Audience Response System (Miller et al. 2003; Table 1). Participants assigned to the control group left the lecture hall without formally making commitments, being told only that they would be contacted by email in the near future. Furthermore, participants in the control group were specifically asked to leave the auditorium after lecture session, and had no knowledge of the intervention group's tasks.

Table 1.  Commitments to change

Commitments
1 I will include anxiety in my differential diagnosis more frequently
2 I will evaluate patients with chronic somatic complaints for anxiety disorders
3 I will remind patients that treatment for anxiety improves, but does not CURE, and use the analogy: eyeglasses correct, they do not cure
4 When I am concerned about PTSD, I will ask: ever had any experience that was so frightening, horrible, or upsetting that, in the past month, you have had nightmares about it or thought about it when you did not want to?
5 I will refer patients to the PTSD resource: http://www.ptsd.va.gov
6 I will remember that cognitive behavioral therapy acknowledges the interplay of THOUGHTS, FEELINGS, PHYSIOLOGY, and BEHAVIOR
7 I will refer patients to cognitive behavioral therapy therapists more frequently
7 I will consider using short-term benzodiazepines and/or beta blockers plus an SSRI for anxiety to improve symptom control

This study had approval from the Institutional Review Board at the University of Massachusetts Medical School.

Design

Seven business days following the lecture, the first follow-up email was sent to both the intervention and control groups. The intervention group's email reminded participants of the commitments they had selected and asked them to self-report whether or not they had “began to implement,” were “planning to implement” or “decided not to implement” (Figure 1) changes in their practice behaviors related to those CTCs. The control group's email asked participants to self-report any changes in their practice behaviors as a result of the lecture. Both were questioned about the confidence in, and barriers to, behavior change. Additionally, follow-up emails were sent to both groups at 30 days post-lecture reminding participants of their commitments and asking them to report whether they had “begun to” were “planning to” or “decided not to” implement practice changes.

Figure 1. The 7- and 30-day post-lecture email reminders.

Measure

For each CTC behavior indicated, level of implementation (“begun to,” “planning to” or “decided not to”) was measured at approximately 7 and 30 days post-lecture. Additionally, two learning-related variables were measured at 7 days post-lecture: (1) a confidence measure querying the level of confidence participants felt that attending the lecture would change their clinical practice and (2) a second measure querying participants on how challenging they believed it would be to integrate information from the lecture into their clinical practice (Figure 2).

Figure 2. Confidence measure.

Analyses

A one-way ANOVA test of variance was used to determine differences between the control and intervention groups with respect to the learning-related variables of confidence and level of anticipated challenges. Test of proportions were used to assess differences between control and intervention groups for the following variables: (1) participants who made commitments to change; (2) participants who intended to, or began to implement change in their practices; and (3) participants who developed from intending to actually implementing changes.

Results

Participants

Out of approximately 800 attendees of the lecture, 144 elected to enroll in the study. Eighty participants (55%) were randomly assigned to the intervention group and 64 participants (45%) to the control group. There were a total of 27 males (19%) and 117 females (81%) dispersed across the two groups. About 50 of those participants were physicians (35%) and 94 of the participants non-physician level clinicians (65%). There were no differences between the intervention and control groups with respect to these demographics (Figure 3).

Figure 3. Study design: enrollment and allocation.

Findings

A total of 97 participants responded to the learning-related variables questionnaire, 57 in the intervention group and 40 in the control group. No significant differences were found between the control and intervention groups with respect to the learning-related variables at 7 days post-lecture.

About 91% (73) of participants in the intervention group and 32% (21) of participants in the control group made a CTC their practice behavior at 7 days post-lecture. The average number of commitments selected by participants in the intervention group was 2.48 (minimum = 1, maximum = 3). The average number of commitments made by participants in the control group was 1.85 (minimum = 1, maximum = 6). The proportion of participants making commitments was greater for those in the intervention group than those in the control group (z = 7.32, p < 0.001).

At 30 days post-lecture, more participants in the intervention group (46 participants) relative to the control group (14 participants) reported contemplating or implementing changes in their practice behaviors (58% vs. 22%, z = 3.74, p < 0.01). When contemplation was excluded and only implementation of changes only was considered, more participants in the intervention group (39 participants) relative to the control group (12 participants) reported change (49% vs. 19%, z = 3.74, p < 0.01). More participants in the intervention group (15 participants) reported moving from contemplating to implementing changes than did those in the control group (2 participants) (19% vs. 3%, z = 2.89, p < 0.01).

Once committed, there were no differences between the intervention group (46 participants) and control group (14 participants) when reporting changes in their practice behaviors (63% vs. 67%, z = −0.30, p = 0.38).

Discussion

The present findings strongly suggest that CTC processes are more likely to induce behavioral changes that persist over time than those activities without CTC. The findings also suggest CTCs can be based upon a predefined set of suggested changes and still have a significant impact on reported behavior change.

This study's findings further add to the evidence already in the literature that spontaneously generated commitments lead to self-reported change (Mazmanian et al. 1998, 2001). Where some studies have been challenged by the lack of control group (White et al. 2004) and sample sizes (Pereles et al. 1997; Wakefield et al. 2003), this study's inclusion of a control group and sufficient sample size (144 participants) make the findings more substantial. The findings in this study differ from those found by Wakefield et al. (2003) where participants who made their own CTCs were more likely to implement those changes in their practice. Therefore, this study adds to the literature which up to this point did not consistently demonstrate the superiority of predefined lecture-derived commitments versus those generated spontaneously.

Potential limitations of this study design include the challenges with recruitment prior to the educational event, sampling, and unique identification of each learner to their audience response tool. With regard to issues with the recruitment and study sample, an uneven number of participants were entered into the control (64) and intervention (80) groups because: (1) not all of the participants who were initially randomized for the study actually attended the lecture; (2) some who were randomized left without enrolling; and (3) several recruiters registered attendees after the lecture at separate stations. Another potential limitation that can be seen is that although the participants were randomized to study groups, this sample included 19% male and 81% female across both intervention and control groups. Future studies should examine the potential influence of gender on CTC.

To address the recruitment and samples issues would necessitate engraining CTC into the culture of CME, providing clear instruction in the process repeatedly during recruitment, and offering the process over many lectures. Studies of the relationship between reported and observed behaviors should focus on behaviors that can be observed asynchronously such as those documented in charts or prescriptions. For example, assume that a learner commits to measure Hbg A1C twice a year in his/her diabetes patients. A comparison could be made of the previous year to the year following the commitment to determine if this commitment has been reached.

CTC processes can ideally be adapted to a large-audience CME lecture through the use of handheld technologies like Audience Response Systems which have demonstrated success in motivating large audiences (Miller et al. 2003). With rapid training, a learner can develop a new paradigm to their education. Furthermore, the database developed from each learner's commitments provides a personalized needs assessment of the specific behavior changes that learner must address to improve the care of their patients. Using ARS in this instance meant linking each tool to the specific responses of the user which required extra time of the participants.

There are also potential limitations related to the value of using self-reported change as an outcome. Further research must be implemented on the relationship between reported and observed changes. A strongly positive relationship would reinforce the underlying assumption that adult learners are reliable reporters.

As technology evolves, most challenges to implementation of CTC into CME activities will be more easily overcome and a further delineation of the efficacy of CTC will occur. From the supposition that all continuing educational components will need to demonstrate behavior change to qualify as an accredited educational event, effective application of CTC to large audience could help to achieve this goal. The success of its future depends greatly on the appropriate recruitment of learners and speakers, and the advance of simple handheld technology as well as the ability to provide prompt and directed feedback to learners that enables them to better self-assess and continue to positively evolve their clinical practices based on the CME they receive.

Expanding CTC processes to all lectures at an event will require the acceptance and participation of all of the speakers. This will depend on effective recruitment of these speakers and clear explanation of how the process works, and it must not be seen as an additional burden. Providing presenters with a report of commitments related to their lectures could potentially enhance the quality of future presentations. If a majority of the audience identified a behavior change in one area, this knowledge could serve as a quality improvement method, thereby enhancing future iterations of that lecture. Commitments could serve as the centerpiece of the learner's educational portfolio and educational plan as well. It is conceivable that they could become a part of the maintenance of certification process, where the attendee measures their practice retrospective to the lecture, and then follows their performance prospectively.

Conclusion

The study demonstrates that CTC can be used effectively and efficiently in large group CME to enhance learning outcomes.

Acknowledgments

The authors wish to thank Heather-Lyn Haley, Erica Sundt, and Karen Rayla of the Department of Family Medicine and Community Health at the University of Massachusetts Medical School for their extensive assistance and development of this project, Donna Abbott and the staff at Harvard Medical School's Department of Continuing Medical Education, Kristin Fludder at pmiCME, and Angela McGill at M-C Communications, LLC. Jenna Clous and the staff at PCIPRO, Inc., and John Horcher. The Current Clinical Issues in Primary Care conference only provides presentations with no direct support from industry. This program is implemented by M-C Communications, an organization which obtains grant funding from several sources including the pharmaceutical industry. M-C funds the Harvard CME office to develop and deliver for the program. Neither M-C Communications nor Industry provide input into the planning of the course, course content, or speakers.

Declaration of interest: The authors of this article have no declarations of interest to report.