The impact of a quality improvement continuing medical education intervention on physicians’ vaccination practice: a controlled study

ABSTRACT

This study investigated the impact of a longitudinal quality improvement continuing medical education (CME) intervention on influenza and pneumococcal vaccination rates for patient populations at high-risk or aged ≥ 65. An observational cohort design with a propensity score to adjust for vaccine eligibility between the intervention and control cohorts was utilized to assess the impact of the intervention among primary care physicians. The intervention was a three-stage quality improvement initiative with CME learning activities. Stage A was an assessment of practice to establish baseline performance. Stage B was participation in learning interventions and individualized action planning for practice change, and Stage C was practice reassessment. Data were also collected for a control group of clinicians who did not participate during the same period. One hundred primary care physicians completed all 3 intervention stages10/14 – 7/15. Altogether, 361,528 patient records of vaccine receipt were compared for those physicians who completed the educational intervention and those who did not. The percentage of physicians’ adult patients receiving influenza or pneumococcal vaccination increased on all measures. The difference between intervention versus control groups was 3.4% higher for influenza ≥ 65 years, 2.1% for influenza high-risk, 0.6% for pneumococcal ≥ 65 years, and 1.4% for pneumococcal high-risk. These results show that physician participation in a quality improvement CME initiative can be an effective strategy to improve vaccination administration. The findings strengthen the evidence that CME learning interventions can advance quality improvement goals and more favorably affect physicians’ practice when educational strategies are utilized.

KEYWORDS:

• Vaccination
• influenza
• pneumococcal disease
• quality improvement
• continuing medical education
• performance improvement

Introduction

Vaccination against influenza and pneumococcal infections has been shown to confer clinical, economic, and public health benefits.^1–6 As a result, various recommendations and guidelines, including the National Foundation for Infectious Diseases and Centers for Disease Control and Prevention, recommend influenza and pneumococcal vaccination broadly, including for adults meeting age requirements or having additional risk factors.^7,8 Despite this evidence of efficacy and benefits and the broad recommendations to administer these and other vaccinations, compliance with these practices is incomplete and more effective strategies for educating public and healthcare providers are needed.⁹ Indeed, various strategies have been undertaken to enhance healthcare providers’ compliance with administering these vaccinations, including teaching medical students and graduate medical education trainees, generating guidelines, and offering continuing medical education (CME) activities. Despite such CME activities and available evidence that CME can favorably change physicians’ practice, the impact of CME to effect favorable change in vaccination practices and to improve organizational performance and community health is uncertain and invites more attention.^10,11 Specifically, more evidence is needed regarding whether physicians’ participation in CME interventions regarding vaccination practices improves vaccination rates.

In this context, the current observational cohort study was undertaken to assess the impact of a CME intervention on vaccination rates in patients who were under the care of participating physicians. The specific intervention was a quality improvement (QI) CME initiative that leveraged the model for improvement framework through performance improvement CME.^12,13 In particular, by tracking vaccination administration in an electronic medical record (EMR), this study compares the rate of vaccine administration to their patients among physicians who participated in a recently completed longitudinal QI CME intervention to the rate administered to a control group of patients of matched physicians who did not participate in the CME intervention over the same secular interval.

Methods

The Cleveland Clinic Center for Continuing Education implemented a quality improvement education intervention with the intent to improve vaccination rates to patients of primary care physicians. An observational cohort design was used to assess the impact of the quality improvement CME learning initiative on vaccination rates among patients cared for by intervention and control physician groups.

The study was deemed exempt from the need for patient consent by the Cleveland Clinic Institutional Review Board.

Study intervention

Primary care physicians employed by Cleveland Clinic in Cleveland, Ohio and affiliated, non-employed primary care physicians who are members of a regional Quality Alliance program self-selected to participate in a 3-stage longitudinal learning intervention, consisting of:

Stage A (October 2014) – Baseline assessment and reflection by the participating physician of her/his current practice regarding influenza and pneumococcal vaccination from performance data captured in the EMR from the prior two years (March 2011- March 2013). Such data included comparison with vaccination rates of other physicians. The purpose of Stage A was to allow the participating physician to formulate a plan for improvement for the upcoming influenza seasons. These data were delivered in a web-based portal which also served as a platform for tracking participation, action plans, and reflection on practice over all three stages of the learning intervention.

Stage B (October 2014-March 2015) – Participation in a variety of learning interventions (e.g., guideline reviews, on-line educational modules regarding the impact of these vaccine-preventable diseases, and official society recommendations for vaccinations) and action planning for improving vaccine practice. Specifically, participating physicians reviewed a set of self-directed learning interventions and resources, and were offered the option to design an action plan for improvement which included the identification of barriers, responsible parties, and benchmarks to achieve improvement goals. The web-based portal included various resources:

• Centers for Disease Control (CDC) and Advisory Committee on Immunization Practices (ACIP) recommended vaccination schedules,¹⁴

• Community Health Surveillance data from the Ohio Department of Health on Influenza Activity in the State of Ohio and surrounding region,¹⁵

• Patient education materials such as the Immunization Action Coalition Handouts for Patients.¹⁶

• Three 30-minute learning activities specifically developed for this activity, entitled:

  • Preventing Pneumococcal Disease in Your High-Risk and Older Patients

  • Influenza Prevention: The 2014–2015 Season and Beyond

  • Developing an Action Plan for Your Practice

Stage C (July 2015) – Re-assessment of vaccination practices to determine outcomes and future improvement needs. In this stage, participating physicians received updated performance data (from March 2014-March 2015) about their own vaccination practices after a 6-month interval. Participants viewed their post-intervention performance, responded to additional reflection questions, and completed an activity evaluation. Reflection questions addressed participating physicians’ insights into their practice, including barriers encountered and changes implemented. Goal-setting for further enhancing vaccination practice was encouraged.

The 3-stage learning interventions and portal were designed and administered as a collaboration between the Academy for Continued Healthcare Learning (ACHL) and the Cleveland Clinic Quality Alliance (QA) and Center for Continuing Education (CCCCE). ACHL led the instructional design, faculty management, coordination of all interventions and education, and development of the activity portal. QA managed overall design of the intervention, participant recruitment, and data extraction. CCCCE helped design the activity, provided overall oversight, led ongoing strategy calls among the partners and, as an Accreditation Council for Continuing Medical Education (ACCME)-accredited provider, implemented certification for CME and Maintenance of Certification (MOC) Part IV credit.

In order to control for usual physician practice and to allow calculation of a propensity score for patients to receive vaccinations (see below), the study used an observational cohort design to assess the impact of the aforementioned quality improvement CME learning intervention on rates of appropriate vaccination receipt among patients of the compared physician groups. The propensity score estimates the effect of the intervention within the observational study and is a method to reduce selection bias.^17,18

Participating physicians

Eligible physicians who were invited to participate included 298 full-time primary care physicians at Cleveland Clinic (CCF), located in Northeast Ohio, as well as members of the Cleveland Clinic Quality Alliance, a program offered to non-employed physicians who admitted to Cleveland Clinic hospitals and affiliates (e.g., the Buffalo Medical Group [BMG]).

The control group consisted of physicians from the invited participant group who did not complete the educational intervention.

As an incentive to participate, physicians were offered up to 20 American Medical Association Physicians Recognition Award (AMA PRA) Category 1 credits and/or MOC Part IV credit for completing all stages.

Outcome measures and eligible patients

The primary outcome measure for the study was the rate with which the participating physician’s patients received the influenza and pneumococcal vaccines in compliance with guideline-based recommendations. The rates were calculated for four specific patient groups from October 2013 to March 2015; these intervals intentionally included periods when seasonal vaccines are available and recommended. The four patient groups were:

1. The percentage of patients aged ≥ 65 who were documented to receive their seasonal influenza annually. This outcome measure was based on the “National Quality Forum (NQF) 0039, Flu Shots for Adults Ages 50 and Over.”¹⁹

2. The percentage of high-risk patients aged 18–64 who were documented to receive their seasonal influenza annually. This outcome measure was based on “Healthy People 2020 goal IID-12.6.”²⁰

3. The percentage of patients aged ≥ 65 who were documented to have ever received the pneumococcal vaccine. This outcome measure was based on “NQF 0043, Pneumococcal Vaccination Status for Older Adults.”¹⁹

4. The percentage of high-risk patients aged 18–64 documented to have ever received the pneumococcal immunization. This outcome measure was based on “NQF 0617, High Risk for Pneumococcal Disease – Pneumococcal Vaccination.”²¹

Electronic medical record (EPIC, Verona, WI) vaccination records from patients of study physicians were eligible for study inclusion if they fit one of the four above criteria and were seen by a study primary care physician within two preceding years (March 2011 – March 2013). The baseline was calculated based on eligible patients in the Cleveland Clinic Health System who received recommended vaccination in the prior period and then compared performance levels in the intervention period based on the same calculation of overall vaccination rate. The outcome measures were based on overall vaccination rates and not based on vaccination results for the same individual patients over each period; thus, results were unaffected by individual patients’ departures from the system or switching providers. Patients with a contraindication to influenza or pneumococcal vaccination were excluded. For the subset of patients cared for by participating Quality Alliance (QA) physicians, vaccine receipt was counted if the medical record documented receipt of a vaccination by a healthcare professional, even if administered outside of the QA Network.

In keeping with available guidelines, patients were deemed to be “high-risk” if the EMR indicated the presence of any of the following conditions:^19–21

• Diagnosis of congestive heart failure, cardiomyopathy, myocardial infarction, angina, or arrhythmia,

• Diagnosis of chronic lung disease (i.e., chronic obstructive pulmonary disease, chronic bronchitis, emphysema, or asthma),

• Diagnosis of diabetes mellitus (excluded if steroid-induced or gestational diabetes),

• Diagnosis of chronic liver disease (i.e., cirrhosis, hepatitis B, or hepatitis C),

• Diagnosis of human immunodeficiency virus infection,

• Diagnosis of renal disease (chronic kidney disease or moderate-to-severe renal disease).

Statistical analysis

To adjust for vaccine eligibility, a propensity score (matching test and control subjects 1:1) was generated to adjust potential differences in test and control subjects related to age, gender, and risk group allocation. A 1:1 propensity-score test-control matched analysis was performed to account for inter-group biases. For propensity-score matching, a logistic regression model was generated on variables significantly different (p < .05) on univariate analysis between test and control groups. Matched variables included age, sex, congestive heart failure, chronic lung disease, chronic liver disease, HIV infection, renal disease and diabetes mellitus. A propensity score from 0 to 1 was generated from this model and assigned to each subject. A nearest neighbor 1:1 variable ratio with propensity scores that fell within a caliper of 0.05 was then used to generate matched cohorts hypothesized to be balanced on potentially confounding baseline characteristics. In the 1:1 matched cohort, stage differences in vaccination rates were generated separately for test and control subjects and were tested with a Chi-square test.

A logistic regression model was performed and the odds ratio and 95% confidence interval (CI) of the odds ratio were analyzed within intervention stages (Tables 1 and 2). An odds ratio >1 indicates a greater likelihood of vaccination for the test group versus the control group. The odds ratios were compared between the baseline and full intervention stages in the matched pair cohort and tested for statistical significance utilizing the Breslow-Day test for homogeneity of the odds ratio (Table 3).

Table 1. Number of patient records analyzed

Measure	Total Baseline (Stage A)	Total Stage C
Influenza: patients aged ≥ 65 years	98,064 Test: 49,032 Control: 49,032	104,906 Test: 52,453 Control: 52,453
Influenza: high-risk patients aged 18-64	73,028 Test: 36,514 Control: 36,514	75,890 Test: 37,945 Control: 37,945
Pneumococcal: patients aged ≥ 65 years	98,084 Test: 49,042 Control: 49,042	104,758 Test: 52,379 Control: 52,379
Pneumococcal: high-risk patients aged 18-64	73,130 Test: 36,565 Control: 36,565	75,974 Test: 37,987 Control: 37,987

Patient records analyzed in the test and control group. A propensity score 1:1 matched Test:Control cohort analysis was performed resulting in the same number of patient records analyzed from the test and control group in each stage.

Table 2. Vaccination rates at baseline (Stage A) and after the intervention (Stage C) for influenza and pneumococcal vaccination in those aged ≥ 65 years and with high-risk factors

OUTCOME
	Influenza 65+			Influenza High-Risk			Pneumococcal 65+			Pneumococcal High- Risk
GROUP	Baseline (Stage A)	Stage C	P	Baseline (Stage A)	Stage C	P	Baseline (Stage A)	Stage C	P	Baseline (Stage A)	Stage C	P
Intervention Group	56.2%	58.7%	<0.001	38.6%	40.4%	<0.001	80.6%	82.7%	<0.001	40.4%	43.8%	<0.001
Control Group	39.3%	38.4%	= 0.003	28.7%	28.4%	= 0.347	56.7%	58.2%	<0.001	28.5%	30.5%	<0.001
Difference Intervention vs. Control, P	3.4 <0.001			2.1 <0.001			0.6 <0.001			1.4 <0.059

Table 3. Impact of the intervention on the vaccination rates odds ratio associated to baseline and completed stages

Vaccine	Population	Baseline (Stage A) Odds Ratio (95% CI)	Stage C Completers Odds Ratio (95% CI)	p Value
Influenza	>65 years	1.98 (1.93–2.03)	2.27 (2.22–2.33)	<0.0001*
	High-risk	1.56 (1.51–1.61)	1.70 (1.65–1.76)	<0.0001*
Pneumonia	>65 years	3.16 (3.07–3.25)	3.43 (3.34–3.53)	<0.0001*
	High-risk	1.70 (1.65–1.75)	1.77 (1.72–1.83)	NS

*statistically significant by Breslow-Day test for homogeneity of the odds ratio (p < 0.05); CI = confidence interval; NS = not significant

For all summaries and statistical analysis, results are presented separately within the four subgroups identified by age and risk categories.

Statistical analysis was conducted using SAS version 9.4 (Cary, NC).

Results

A total of 298 physicians from the Cleveland Clinic Quality Alliance network were invited to participate; 273 (91.6%) elected to participate in at least the first stage. Of these, 135 (BMG [n = 8], CCF [n = 113], independent [n = 14]) moved through Stage B (test group) and 100 (BMG [n = 4], CCF [n = 87], independent [n = 9]) moved through Stage C, completing the entire learning intervention.

Table 1 shows the number of patient records that were analyzed. Table 2 shows the differences in the rates of vaccinations administered between the intervention and the control groups of physicians by specific patient outcome group. Also shown in Table 2 are changes in vaccination rates from baseline to Stage C within the intervention and control groups.

In assessing odds ratio of vaccine receipt after the learning intervention, there were significant increases in three of the four measures (P< .0001; Table 3).

Regarding changes from baseline to Stage C within the intervention and control groups (Table 2), the intervention group of physicians achieved statistically significantly higher rates of guideline-compliant influenza vaccination from baseline (Stage A) to Stage C in several outcome groups, i.e., the percentage of patients ≥ 65 years documented to have received the seasonal influenza vaccine in the intervention group increased from 56.2% at Stage A to 58.7% at Stage C (P< .001; Table 2). Also, rates for seasonal influenza vaccination in high-risk patients aged 18–64 years increased in the intervention group (38.6% at Stage A vs. 40.4% at Stage C; P< .001; Table 2).

In contrast, no significant changes from baseline to Stage C were observed in vaccination rates for other outcome groups – those with HIV, renal disease, and COPD. Still, it is noteworthy that the percentage of high-risk patients who had received the influenza vaccine at Stage A was highest in patients with human immunodeficiency virus (56% at Stage A) and renal disease (55% at Stage A); similar vaccination rates in these groups were also observed in Stage C. Rates were lowest in patients with chronic lung disease (39% at Stage A), with a non-significant increase observed in Stage C (41%).

Regarding the rate of pneumococcal vaccination for patients of intervention group physicians from baseline to Stage C, significant changes were also observed. Specifically, the percentage of patients ≥ 65 years documented to have ever received a pneumococcal vaccine increased in the intervention (80.6% at Stage A vs. 82.7% at Stage C; P< .001) and control (56.7% at Stage A vs. 58.2% at Stage C; P< .001) groups (Table 2). Pneumococcal vaccination rates in high-risk adults also increased in the intervention (40.4% at Stage A vs. 43.8% at Stage C; P< .001) and control (28.5% at Stage A vs. 30.5% at Stage C; P< .001) groups (Table 2). The percentage of high-risk patients who were vaccinated against pneumococcal disease at Stage A and C was highest in patients with human immunodeficiency virus (66% at Stage A and 71% at Stage C) and diabetes mellitus (66% at Stage A and 68% at Stage C). Rates were lowest in patients with coronary heart disease (35% at Stage A) with an increase in rates of vaccination at Stage C (38%).

At the end of the activity, 83% of participating physicians self-reported interpreting their performance data to assess the impact of the educational interventions. Furthermore, 88% reported that they had worked with team members, such as nurses and physician assistants, to implement interventions, and 72% reported having made appropriate process adjustments in their improvement efforts.

Participating physicians in the intervention group self-reported process changes and general improvements to care as a result of completing the CME activity. These qualitative data included the following: improved vaccination rates, system-based improvements that led to fewer hospital admissions; better data collection efforts; more efficient work flow; and heightened clinical awareness about vaccination practice.

Discussion

In this observational cohort study of an intervention to enhance vaccination practices by primary care physicians, the main finding is that vaccination rates among patients cared for by physicians in the intervention group increased modestly but significantly more than in patients of control group physicians regarding three of the four primary vaccination rate outcomes. Specifically, the propensity score analysis showed that the likelihood of receiving a vaccine was greater in the intervention group compared with the control group for both influenza measures and for the high-risk pneumococcal vaccination group. Also, the percentage of adults who received an influenza or pneumococcal vaccination increased from baseline on all measures in the intervention group. In the context of significantly improved rates in the intervention group, vaccination rates also increased for two measures in the control group. It is noteworthy that the control group performed worse for the subset of high-risk patients and those ≥ 65 years (Table 2) over the study period (influenza: 39.3% at Stage A to 38.4% at Stage C; P = .003 and pneumococcal vaccination: 28.7% at Stage A to 28.4% at Stage C; P = .347). This difference between the intervention and control group rates strengthens this proof-of-principle observation that the CME intervention (Stage B) as designed can enhance vaccination receipt among patients of primary care physicians.

The magnitude of increase in vaccination rates observed in the intervention group were small but, as noted, significant, i.e., 0.6% to 3.4% (Table 2). Still, these small rate increases translate to large numbers of patients affected. For example, a 1% change in the intervention group affects 1,807 patients. The enhanced rate in the intervention group in this study translates to an additional 3,426 patients vaccinated. This substantial increase in the number of vaccinated patients was achieved at an implementation cost with the QA, ACHL and CCCCE of $389,325, or the modest cost of $114 per patient.

Notably, while the overall intervention was effective, some specific patient subsets experienced no change, including some high-risk populations (i.e., those with HIV, renal disease, and COPD). This lack of change suggests that, not surprisingly, factors outside of the providers’ participation in this initiative also affected vaccination practices. Examples might include patients’ aversion to receive vaccines and motivation or characteristics unique to these patient subsets.

These findings extend available knowledge in several ways. First, to our knowledge, while there are a few studies examining quality improvement models directed at enhancing vaccine practice, research examining the impact of quality improvement enhanced by CME for improving vaccination practices is sparse. For example, QI models have leveraged process changes to prompt compliance such as using point-of-care prompts and algorithms, or teaching QI principles to caregivers for application to local workflows where improvements are needed.^22–26 For example, Gilkey et al. showed that education with assessment and feedback in a localized QI intervention for HPV vaccination improved vaccination rates, while also emphasizing the need for further implementation research.²⁶ Building on that observation, the current study demonstrates that longitudinal community-based QI interventions paired with CME can be associated with enhanced vaccination rates.

Second, going beyond the specific context of vaccination practices, this study adds to a growing body of literature showing that CME-based interventions that combine education with continuous quality improvement initiatives can enhance clinical outcomes. The study also demonstrates utilizing EMR-based data to enhance physician practice as part of a CME intervention. Effectiveness in achieving outcomes can originate from education design based and focused on continuous quality improvement and managing healthcare professional learning experiences within the context of improvement goals for practice and systems contexts.^27–33 By measuring clinical outcomes from EMR data in the patients actually cared for by the participating physicians in the study, this study strengthens the evidence that CME can favorably affect physician practice when designed by these standards.

At the same time, several limitations of the study warrant discussion. First, the study was an observational cohort study rather than a randomized trial; as such, physicians self-selected participation in the learning interventions, creating the possibility of selection bias. As a specific example of how such selection bias might confound the study results, that baseline vaccination rates were higher in the intervention group likely reflects study participation by the most motivated physicians. Similarly, despite propensity matching, providers in the control group may have cared for patients who were less inclined to receive vaccination.

Another limitation of the study is that the available data were confined to fields that were available in the EMR. As a result, not all of the high-risk factors outlined by Advisory Committee on Immunization Practices were captured as discrete data elements in our analysis (e.g., cochlear implants, asplenia). Further, neither specification of the specific type of pneumococcal vaccination administered (i.e., PCV13 vs. PPSV23) nor documentation of vaccination administration in non-traditional settings (i.e., in a pharmacy) was captured.

A third study limitation was the rate of attrition of providers across the three study interventions, i.e., only 34% of physicians completed all three intervention stages. While most of the invited target audience opted in (92%), 63% dropped off after the first stage. This attrition may have caused vaccination rates among the intervention group to be underestimated but also clearly indicates the need to enhance study retention for participating physicians. While we imagined that the opportunity for participants to receive feedback on their patients’ vaccination rates and receiving CME and MOC credits would incent full participation, the attrition rates in this study highlight the need to develop additional retention incentives.

Finally, because the primary study outcome was vaccination receipt by patients irrespective of which provider ordered the vaccine, attribution of vaccination orders to the participating study physicians is somewhat uncertain. We nonetheless suspect that most of the vaccination orders for patients which were measured in the study were placed by study physicians or by providers in their practices, who were likely influenced by the participating physicians.

In summary, the current study shows that physician participation in a quality improvement CME activity can be associated with enhanced rates of vaccination receipt by their patients. The study strengthens the evidence that CME learning interventions can enhance vaccination practices and provides further proof-of-principle that CME-based learning interventions can favorably affect physicians’ practice.

Disclosure of potential conflicts of interest

No potential conflict of interest is reported by the authors.

Ethical approval

Submitted to IRB for approval and an exemption was granted for being an internal quality assessment and improvement activity.

Acknowledgments

The authors would like to acknowledge those who contributed to planning this activity, including; T. Elsawy, MD, T. Nadas, MBA, formerly of the QA; Mark Van Buskirk of Data Reduction; and Lisa Keckich, MS of ACHL; Susan Rehm, MD; and William Schaffner, MD.

Supplementary material

Supplemental data for this article can be accessed online at http://doi.org/10.1080/21645515.2020.1737457.

Additional information

Funding

This activity was supported by an educational grant from Pfizer, Inc. [Grant Number: 9714885]. The Cleveland Clinic Foundation Center for Continuing Education and the Academy for Continued Healthcare Learning staff members have no relevant affiliations or financial relationships to disclose