Evidence-based vaccination strategies in obstetrics and gynecology settings: Current practices and methods for assessment

Abstract

Obstetrician-gynecologists have the potential to play an important role in the delivery of immunizations to women. However, despite national recommendations, immunization rates among pregnant women and adults in general remain low. Pragmatic immunization delivery trials are needed to demonstrate how best to deliver vaccines in such settings. We report the development and implementation of 2 novel methodologies for immunization delivery research and quality improvement in such settings. The first was the development and application of a 47-point Immunization Delivery Scale that formally assessed variability among practices in their engagement in a variety of evidence-based practices for improving immunization rates. The second was a covariate-constrained randomization technique – a method for achieving balance between study arms in cluster-randomized trials that is especially applicable to pragmatic trials.. To best achieve meaningful and interpretable findings, we recommend use of these or similar techniques in future immunization research and quality improvement projects in OB/GYN settings.

Keywords:

• immunization
• methods
• ob/gyn
• pragmatic
• pregnancy
• randomization

Introduction

The childhood vaccination program in the United States has been widely hailed as one of the greatest success stories of public health.¹ However, a similar success story cannot be claimed for adult immunization. Despite established guidelines for several vaccines for adults, which include both universal recommendations (influenza, Tdap) and targeted recommendations (HPV, meningococcal, pneumococcal, zoster), vaccination rates in the adult population are uniformly low, particularly when compared to the success of the childhood vaccination program.² OB/GYN providers have the potential to play an important role in delivering vaccinations to this population.

The Task Force on Community Preventive Services has recommended several evidence-based interventions to increase immunization coverage for a variety of populations.^9-12 These strategies include patient reminder/recall, provider alerts, provider assessment and feedback, the use of standing orders, and multi-modal interventions, which can include some combination of reminder/recall, patient education, tracking of vaccination status, standing orders, and provider alerts. However, there are barriers to implementing these strategies in many settings. Reminder/recall, for example, can be both time-consuming and costly.¹³ Also, the bulk of the research on these interventions has concentrated on intervention effectiveness among young children and adults, primarily in pediatric, family medicine, and internal medicine settings, with very little evaluation of such strategies in OB/GYN settings. Thus, there is a need for pragmatic effectiveness trials to assess the feasibility and impact of these recommendations in OB/GYN settings, which can have a significantly different focus and practice structure than primary care.

We describe a study that was undertaken as part of preparatory work for a large, ongoing, pragmatic, cluster-randomized trial among 12 private OB/GYN practices of a multi-modal intervention to increase uptake of tetanus-diphtheria-acellular pertussis vaccine (Tdap), influenza vaccine, and human papillomavirus vaccine (HPV). As with many interventions related to immunization delivery, our intervention was designed to improve immunization rates within practices and was therefore practice-based. Thus in this trial randomization occured at the practice, rather than patient level.

A major limitation to cluster-based pragmatic trials is the small sample size, which reflects the number of practices rather than the number of patients involved in the trial. Because of this, a key issue in cluster-randomized trials is the possibility of covariate imbalance in practices assigned to different treatment arms. Stratified randomization is often performed when assigning study arms in such trials so as to minimize the effects and potential sources of confounding that can vary by practice characteristics (size, urban or rural, etc). However, this still has the potential to result in highly unbalanced groups if the sample size is sufficiently small.¹⁴ For example, relative to studies of immunization delivery, practices may be very different in terms of their baseline activities devoted to immunization at the time of randomization, which could significantly confound any impact of immunization delivery interventions implemented in the trial.

As part of our preparation for the larger, cluster-randomized, pragrmatic trial, we performed a preparatory study with 3 objectives: 1) to describe immunization activities among the 12 private OB/GYN practices involved in the trial; 2) to create an ‘Immunization Delivery Scale’ to assess these practices' baseline and future immunization-delivery efforts; and 3) to use the Immunization Delivery Scale along with other variables to utilize a novel method of randomization, called covariate-constrained randomization, that could be used in future pragmatic immunization delivery trials in OB/GYN, and potentially other, settings.

Results

Practice demographics

There were 12 practices in the study, 8 urban or suburban and 4 rural (Table 1). There were a mean of 5.8 full-time equivalent (FTE) providers per practice (range 2–19) Practices had a mean of 45.9 deliveries per month (range 12–125) or 7.9 deliveries per FTE per month. Overall mean public insurance among the practices was 16.8% (range 3–42%).

Table 1. Characteristics of Participating Practices.

Practice	FTE providers, n	Deliveries per month, n	Deliveries per FTE per month	Public Insurance, %
Urban/Suburban Clinics
Practice 1	19	125	6.6	4%
Practice 2	5	12	2.4	14%
Practice 3	4	30	7.5	5%
Practice 4	4	60	3	13%
Practice 5	9	63	7	3%
Practice 6	6	78	13	32%
Practice 7	4	30	7.5	4%
Practice 8	6	48	8	19%
Rural Clinics
Practice 9	2	33	8.3	42%
Practice 10	3	13	4.3	20%
Practice 11	2	15	7.5	40%
Practice 12	5	40	8	6%
Mean	5.8	45.9	7.9	16.8

Abbreviations: FTE, full-time equivalent.

Baseline immunization practices – assessing immunization delivery scale components

We systematically assessed all components of the Immunization Delivery Scale. All 12 practices stocked HPV vaccine. Six practices stocked influenza vaccine, and 2 stocked Tdap vaccine. Only 2 practices tracked prior vaccinations by assessing the vaccination history and placing it in the EMR, and that was for HPV vaccine only. Few practices were otherwise tracking vaccination history for any of the 3 target vaccines, although most had a field in their EMR to do so. Three practices systematically identified patients eligible for influenza vaccine, and 2 did this for Tdap vaccine. For the majority of practices, however, identification of patients eligible for vaccination was at the discretion of the provider or medical assistant (MA) (i.e. not systematic). Similarly, while most practices offered some form of patient education and educational materials regarding vaccination, this was most often at the discretion of individual providers. Only one practice systematically delivered educational materials to patients for HPV vaccine and one did this for Tdap vaccine (none for influenza). Most offered Vaccine Information Statements (VIS) for HPV vaccine, but only one practice each offered these for Tdap or influenza vaccines. Seven of the 12 practices had a functioning standing order program for influenza vaccine, although only 1 practice had such a program for Tdap, and none for HPV. Systems to reduce missed opportunities for vaccination were uncommon, with only 3 practices having provider prompts or provider education and feedback for HPV vaccine, and only one doing this for influenza vaccine and none for Tdap. Finally, 8 of the practices used some form of reminder/recall for HPV vaccine series completion (none for initiation), with only 2 using reminder/recall for influenza vaccine and none for Tdap.

The immunization delivery scale

Information from the baseline assessment was used to develop composite, and vaccine-specific Immunization Delivery Scores for each practice. Composite Immunization Delivery Scores overall and by practice location are shown in Table 2, and the scoring rubric is shown in Figure 1. The mean overall score was 15.1 – all practices were far from optimized in their immunization delivery activities (range 8–25, out of a possible 47). By vaccine, the highest mean score was for HPV (6.4) followed by influenza (5.8) and Tdap (2.9).

Figure 1. The Immunization Delivery Scale.

Table 2. Composite Immunization Delivery Scores Overall and by Practice Location.

Immunization Delivery Score	Overall Mean (Min, Max)	Urban Mean (Min, Max)	Rural Mean (Min, Max)
Tdap (0–16)	2.9 (0, 9)	2.3 (0, 9)	4.0 (1, 8)
Influenza (0–16)	5.8 (1, 10)	6.1 (2, 8)	5.3 (1, 10)
HPV (0–15)	6.4 (4, 9)	6.4 (4, 9)	6.5 (6, 7)
Immunization Delivery Score^* (0–47)	15.1 (8, 25)	14.8 (8, 22)	15.8 (9, 25)

Abbreviations: Min, minimum; Max, maximum; Tdap, tetanus-diphtheria-acellular pertussis vaccine; HPV, human papillomavirus vaccine.

* Scores of each practice were: 8, 9, 9, 13, 16, 16, 17, 17.5, 20, 22, 23, and 25.

Covariate constrained randomization

In addition to the Immunization Delivery Scale, several factors were identified as influential on practice-based immunization rates, and therefore used as variables in the covariate-constrained randomization process. The randomization allocation was set a priori so that in the final randomization, there would be 4 urban/suburban practices and 2 rural practices in each study arm. There were 924 possible randomizations of 12 practices into 2 groups. Only randomizations with equal numbers of urban and rural practices in each group were considered which resulted in 420 possible randomizations. Differences in group means between optimal randomizations (i.e., top 10% that are the most balanced) and all other randomizations are shown in Table 3. For all of the chosen covariates, the mean differences between study arms for the optimal randomizations were significantly less than the other randomizations.

Table 3. Magnitude⁺ of Differences in Group Means for Optimal Set of Randomizations versus All Other Sets.^*

	Optimal Randomizations		Other Randomizations
Variable	Mean Difference	Min, Max	Mean Difference	Min, Max
Provider FTEs, n	1.7	0.5, 2.7	2.4	0.3, 4.6
Deliveries/month, n	8.0	0.2, 19	15.1	0.2, 42
Public Insurance, %	3.5	0.3, 7.3	6.3	0.0, 19.7
Flu Vaccine Score	0.7	0.2, 1.5	1.5	0.0, 4.5
HPV Vaccine Score	0.3	0.2, 0.8	0.7	0.2, 1.8
Tdap Vaccine Score	1.0	0.3, 2.3	1.6	0.3, 4.1
Immunization Delivery Score	0.9	0.3, 2.3	3.0	0.1, 8.9

Abbreviations: FTE, full time equivalent; Min, minimum; Max, maximum; Tdap, tetanus-diphtheria-acellular pertussis vaccine; HPV, human papillomavirus vaccine.

* p<0.01 for difference between means for optimal randomizations compared to other randomizations on all variables

The final randomization was chosen at random from the set of optimal randomizations and is shown in Table 4. There were no significant differences between intervention and usual care arms in any of the chosen covariates, including number of FTE providers (6.5 versus 5.2), number of deliveries per month (51.5 vs. 39.7), percent of patients with public insurance (17.0 versus 16.7), and the Immunization Delivery Score (14.6 vs. 15.7).

Table 4. Final Randomization for Study Using Covariate-Constrained Randomization Technique.

Variable	Intervention Mean (Min, Max)	Control Mean (Min, Max)	P-Value
Provider FTEs, n	6.5 (2, 19.4)	5.2 (2, 9.2)	0.6563
Deliveries/month, n	51.5 (13, 125)	39.7 (12, 78)	0.5549
Public Insurance, %	17.0 (4, 42)	16.7 (3, 40)	0.9698
Immunization Delivery Score (0–47)	14.6 (8, 20)	15.7 (9, 25)	0.7555

Abbreviations: FTE, full time equivalent; Min, minimum; Max, maximum.

Discussion

With this study, we describe current immunization practices among a group of diverse private OB/GYN practices, and demonstrate great variability in the amount of effort these practices expended on immunization delivery at the time of arm designation for our larger, cluster-randomized, pragmatic clinical trial. For all practices assessed, we found substantial potential for improvement in implementing evidence-based practices for improving vaccination uptake. Formal assessment of these differences was achieved by assigning our Immunization Delivery Score to each practice and vaccine. This score, along with other variables considered influential on practice-based immunization delivery, was successfully incorporated into a covariate-constrained randomization statistical model to develop balanced arms for our pragmatic cluster-randomized immunization delivery trial. Based on these results, we believe that similar techniques should be considered in future immunization delivery trials, as these novel methods were effective in achieving balance between our study arms.

The significant variability we found in immunization activities among our study practices is a cause for concern. Similar to national data, many practices are not following current ACIP recommendations.^15,16 However, immunization delivery in OB/GYN settings is a relatively new paradigm. Prior to the 2009 influenza pandemic, few OB/GYN practices delivered influenza vaccine, with national rates from 11 to 35% for influenza vaccine in pregnancy in the prior decade.^17,18 With the 2009 pandemic, pregnant women were identified as a priority group for vaccination, and there was a concerted and successful public health effort to collaborate with OB/GYN practices to deliver influenza vaccine.¹⁹ Since that time, many, although notably not all, practices have continued to deliver influenza vaccine to their pregnant patients.²⁰ These successes in prior immunization delivery efforts demonstrate the potential reach of OB/GYNs for vaccination. However, further study is needed to understand how and which evidence-based immunization delivery strategies work in this practice setting. While our examination of immunization practices was only in Colorado and therefore not necessarily generalizable, our findings provide a more detailed snapshot of current immunization practices in the OB/GYN setting than has been reported previously.

The use of HPV vaccine, and strategies to increase its uptake, were more prevalent in our study cohort than for influenza or Tdap vaccines. Reasons for this difference are not entirely clear, since the recommendation for influenza vaccine in pregnancy is long-standing and HPV vaccine was introduced in 2006. It may be that OB/GYN providers are more enthusiastic about HPV vaccine because they see the ill effects of HPV disease very commonly. It may also be that some providers are more comfortable delivering HPV vaccine – as opposed to Tdap or HPV vaccines – since it is given outside of pregnancy. Relative to influenza vaccine, it may also be that HPV vaccine is simply an easier vaccine to manage, since it can be given anytime, rather than the more complicated process of delivering influenza vaccine in a seasonal fashion. For Tdap vaccine, it is likely that at the time of our study, its use was limited due to the relatively recent changes in recommendations to give the vaccine during pregnancy. With broadened recommendations for Tdap in every pregnancy, it would be expected that Tdap utilization among these practices currently is higher.

We believe that our Immunization Delivery Scale measure can be a useful tool for future immunization delivery quality improvement and research projects in OB/GYN settings. We envision that our tool may be used in a variety of settings. For example, OB/GYN practices who seek to improve immunization uptake among their patients may take a baseline measurement of their current activities using the tool, design a quality improvement program, and then re-measure their activities using the tool after some period of time and then compare scores pre- and post-intervention. Because actual immunization rates may be difficult to measure for a variety of reasons, a tool such as ours may offer a practice a simple way of demonstrating quality improvement. Similarly, this tool could be used in a health system to assess current immunization practices in each of its clinics, so as to focus improvement efforts in the ones that need the most help. Finally, we are of the opinion that there is a need for dissemination and implementation trials for evidence-based strategies for immunizations in OB/GYN settings, and our Immunization Delivery Score could provide an outcome measure for such studies. It should be noted, however, that this was not a validation study. That is, we are unable in this study to correlate the immunization delivery score with actual vaccination rates. However, there was a broad distribution of scores among the practices in the study, suggesting that the tool can show differences among practices in the implementation of evidence-based strategies known to increase immunization rates. In future work, we will attempt to correlate scores with actual vaccination rates.

In pragmatic cluster-randomized trials such as the one described here, achieving balanced study arms is important because imbalance can limit the ability to draw conclusions about an intervention's effectiveness. Imbalance weakens the case for causal inference, which is the key strength of randomized trials, because differences between study arms after an intervention may not be because of the intervention but rather because of differences in the populations in each arm. The most common methods for achieving balance, stratification and matching, are often insufficient in achieving balance.^14,21 In this current study, we demonstrate a method for achieving such balance on important covariates that could affect the study's outcome. To do this, it is important to note that investigators may need to spend significant time collecting data on the chosen covariates before randomization takes place. Such data collection can be time-consuming, and this should be considered when designing the timeline for grant applications using this type of randomization. Similarly, it is crucial to be thoughtful up front about which covariates to include. While covariate-constrained randomization can accommodate more covariates than simple stratification, including potentially irrelevant covariates can lessen the influence of more important ones. In our study, the covariates were a priori chosen based on the potential for their impact on the study outcome of vaccination rates. We decided that urban and rural practices should be in different strata because of differences in patient populations not measured by the other covariates. The size of a practice and the number of deliveries could impact a practice's vaccination practices and were therefore also chosen as covariates of importance. For example, a practice with few deliveries may be less inclined to stock Tdap vaccine. We included proportion of public insurance as a covariate because of known differences in health-seeking behavior in lower socioeconomic status populations, and because of the possibility of financial issues being a barrier to vaccination. Finally, our Immunization Delivery Scale was able to account for baseline differences in current immunization practices.

There are some important limitations to this study. First, it represents data gathered in one geographic area, and therefore current immunization practices may not represent practices nationwide. Also, our Immunization Delivery Scale, while potentially a useful tool for assessing immunization practices, may not be reflective of actual immunization rates. Finally, regarding the method of covariate-constrained randomization, the procedure is only as good as the pre-randomization data that goes into the model, and such data may be difficult to obtain or inaccurate.

Conclusions

Immunization delivery in OB/GYN settings has not yet reached its full potential. With this study, we suggest 2 useful tools for future cluster-randomized, pragmatic trials within OB/GYN settings, our Immunization Delivery Scale and the method of covariate-constrained randomization. While these techniques require additional effort, applying such methods to future trials has the potential to improve our ability to produce high quality, generalizable study findings from studies performed in real-world settings.

Methods

Study population

This study was performed within 12 private OB/GYN offices (8 urban/suburban, 4 rural) in Colorado. The practices had been previously approached and recruited to participate in a large, cluster-randomized, pragmatic trial to increase vaccination uptake in OB/GYN offices.

Data collection

We created an interview guide to assess practice characteristics and immunization related activities (available upon request). The interview guide was designed by the study team with input from a qualitative research methods expert. There were 7 closed-ended questions regarding practice size and patient characteristics and 8 open-ended questions designed to assess current vaccination operations used by the OB/GYN offices for each vaccine of interest, including influenza, Tdap, and HPV vaccines. Vaccination operations assessed included whether a vaccine was currently in stock, if there was a system for tracking vaccine history, if there was a system for identifying patients eligible for vaccines, if the practice utilized standing orders for vaccination, if there was a system for delivering patient education, if there was a system for avoiding missed opportunities for vaccination, and if the practice utilized reminder/recall.

Interviews were conducted with 1–2 staff members per office who were identified by the office as the most knowledgeable about the practices' immunization program and office characteristics. The interviews took place between December 2011 and March 2012. Interviews were conducted by phone, were audio-recorded, and summarized in detail.

Creation of the immunization delivery scale

We created the Immunization Delivery Scale based on evidence-based practices for increasing immunization rates as put forth by the US Task Force on Community Preventive Services. For OB/GYN practices, we chose to focus on Tdap, HPV, and influenza vaccines. We chose these vaccines as 2 are recommended in pregnancy (Tdap and influenza) and because all study OB/GYN practices delivered HPV vaccine at the time of the study. We also felt that these 3 vaccines were the ones most likely to be delivered by OB/GYNs nationally. We assigned points for each vaccine across 7 categories. The first was whether or not a practice stocked the vaccine of interest, and for Tdap and influenza, if this was just for select patients (such as pregnant patients) or all patients. The other 6 categories were based on recommendations from the US Community Preventive Services Task Force. These included tracking vaccine history, identifying patients eligible for vaccination, offering patient education, having a standing order program, reducing missed opportunities for vaccination, and the use of reminder/recall. The scoring rubric is shown in Figure 1.

The scoring rubric was applied to the practice data, described above. Two members of the study team (STO, JP) independently scored each practice according the rubric. Where there were disagreements in scoring, consensus was reached through discussion with the entire study team.

Covariate-constrained Randomization

To perform covariate-constrained randomization, variables must be identified on which to randomize. These variables are chosen based on 1) that they may be directly associated with the study outcome, 2) being a potential confounder, or 3) that they could affect implementation of the intervention. For this study, we chose to randomize based on the following criteria: the number of FTE providers, the number of deliveries per month, the percent of patients with public insurance, and the Immunization Delivery Score.

The overall concept of covariate-constrained randomization is to generate all possible randomization combinations based on the covariates involved, and identify an optimal set of randomizations that provide the most balance between arms. In our trial, to do this, we first generated all possible randomizations using SAS Proc IML. We then standardized our randomization variables using z-scores, which allowed each covariate to contribute roughly equally to the balancing process. We then generated a file with group means for each randomization and computed a balance criterion for each randomization. This balance criterion was defined as the sum of squared difference between the study groups on our chosen covariates. We then examined the distribution of the balance criterion and set the balance criterion as <0.85, which defined the ‘optimal set.’

β = (#FTEsg1 - #FTEsg2)² + (#deliveriesG1 – #deliveriesG2)² + (%publicG1 – %publicG2)² + (FluScoreG1 - FluScoreG2)² + (HPVscoreG1 - HPVscoreG2)² + (TdapScoreG1 – TdapScoreG2)² + (IzScoreG1 – IzScoreG2)²

This optimal set represented the top 10% of all possible randomizations (i.e. the randomizations with the least mean differences between study arms on the chosen covariates). The final single randomization used to assign practices to intervention or usual care arms was chosen randomly from the optimal set.

All analyses were performed using SAS 9.3 (Cary, North Carolina).

Disclosure of potential conflicts of interest

Amanda Dempsey serves on advisory boards for Merck and Pfizer. She does not receive research funding from either company, nor did either company play any role in this research.

Author contributions

SOL designed the study, analyzed the data and wrote the first draft of the manuscript.

SEB and JP assisted with the study design, collected data, assisted with analysis and wrote sections of the manuscript.

AFD designed the study, analyzed the data and edited the manuscript.

Funding

This work was funded by the Center for Disease Control and Prevention (IP000501–03). The opinions expressed in this manuscript do not necessarily represent those of the funding agency.