The COVID-19 vaccine concerns scale: Development and validation of a new measure

ABSTRACT

Reasons for COVID-19 hesitancy are multi-faceted and tend to differ from those for general vaccine hesitancy. We developed the COVID-19 Vaccine Concerns Scale (CVCS), a self-report measure intended to better understand individuals’ concerns about COVID-19 vaccines. We validated the scale using data from a convenience sample of 2,281 emergency medical services providers, a group of professionals with high occupational COVID-19 risk. Measures included the CVCS items, an adapted Oxford COVID-19 vaccine hesitancy scale, a general vaccine hesitancy scale, demographics, and self-reported COVID-19 vaccination status. The CVCS had high internal consistency reliability (α = .89). A one-factor structure was determined by exploratory and confirmatory factor analyses (EFA and CFA), resulting in a seven-item scale. The model had good fit (X2[14] = 189.26, p < .001; CFI = .95, RMSEA = .11 [.09, .12], NNFI = .93, SRMR = .03). Moderate Pearson correlations with validated scales of general vaccine hesitancy (r = .71 , p < .001; n = 2144) and COVID-19 vaccine hesitancy (r = .82; p < .001; n = 2279) indicated construct validity. The CVCS predicted COVID-19 vaccination status (B =  −2.21, Exp(B) = .11 [95% CI = .09, .13], Nagelkerke R² = .55), indicating criterion-related validity. In sum, the 7-item CVCS is a reliable and valid self-report measure to examine fears and concerns about COVID-19 vaccines. The scale predicts COVID-19 vaccination status and can be used to inform efforts to reduce COVID-19 vaccine hesitancy.

KEYWORDS:

• Vaccine hesitancy
• COVID-19
• scale development
• scale validation
• survey

Vaccination is a critical tool for combating the coronavirus disease 2019 (COVID-19) pandemic, as it can prevent infection, reduce transmission, and minimize severe illness.¹ As of September 2021, despite widespread availability and no costs associated with vaccination, only 76% of U.S. adults had received at least one dose of a COVID-19 vaccine.² Moderate-to-high levels of COVID-19 vaccine hesitancy have also been noted in many other countries, including but not limited to Russia,³ Australia,⁴ Poland,⁵ and Japan.⁶

The need to understand vaccine hesitancy is becoming increasingly important in light of the evolving situation related to the COVID-19 virus and vaccine administration. First, the evolving nature of the virus continues to underscore the importance of widespread vaccination. The COVID-19 vaccines available are highly effective at preventing severe disease^7,8although breakthrough infections are possible. Even with breakthrough cases, vaccination is believed to be beneficial by reducing the time over which an infected individual is contagious, thus decreasing the risk of spreading infection to others.⁸ Second, the introduction of booster shots, supported by evidence that protection from a COVID-19 vaccine declines over time,⁹ will be critical to ensure protection is maintained. Booster shots for some of the COVID-19 vaccines are currently being suggested to be administered six months after the initial dose.¹⁰ Third, the increasing use of mandates that require vaccination, recently bolstered in the U.S. by Food and Drug Administration (FDA) approval of the Comirnaty BNT162b2 (mRNA) vaccine,¹¹ means many vaccine-hesitant individuals may now be faced with decisions about getting vaccinated that will affect their employment or education.

Based on these important public health issues, understanding the drivers of vaccine hesitancy is important to enhance immunizations. Research studies have revealed concerns about safety, misinformation, lack of confidence, lack of trust, and politicization.^12,13 While some of these themes are similar for the broad topic of vaccinations, evidence suggests that there are differences between general vaccine hesitancy and COVID-19 vaccine hesitancy. Specifically, drivers of COVID-19 vaccine hesitancy include unique factors such as endorsement of COVID-19 conspiracy theories, including COVID-19 being a biological weapon, a way to keep citizens in line, or part of a bigger plot,^14,15 or that the vaccine contains microchips.¹⁴ Thus, current measures of general vaccine hesitancy are not sufficient to study this issue. Prior work has led to the development of COVID-19 vaccine hesitancy measures,^16–19 which examine willingness to get, and perceived importance of, the COVID-19 vaccine. However, while these measures assess one’s hesitancy to get vaccinated, they do not provide an understanding of the root causes of COVID-19 vaccine hesitancy. Some recent scales have been developed that include items to examine contributors to COVID-19 vaccine hesitancy,^{20, 21} but thus far, items in these scales tend to be those that are common to general vaccine hesitancy (e.g., adverse reactions, side effects) and not reflective of the nuanced hesitancy that surrounds COVID-19 vaccines.

For all of these pertinent and pressing reasons, addressing COVID-19 vaccine hesitancy is critical to increase vaccination rates. To do so, we must understand the sources and motivations of hesitancy to effectively address it with strategies such as targeted messaging and education. Being able to identify and measure COVID-19 vaccine concerns is the first step toward developing these strategies. Therefore, the goal of this study was to develop and validate a new measure for COVID-19 vaccine concerns.

Materials and methods

Participants

This study was an evaluation of vaccine hesitancy among U.S. emergency medical services (EMS) healthcare professionals. This specific population was chosen as EMS professionals are frontline providers in the health-care system who regularly enter homes and interface with the public. Thus, this population is at high risk for public exposure to COVID-19, and also need to be immunized to minimize public spread of disease. Participants ranged in age from 18 to 83 years old and were recruited from the National Registry of Emergency Medical Technicians’ (National Registry) database, which contains contact information for approximately 420,000 EMS professionals in the U.S. This evaluation is based on a larger study conducted to determine vaccine hesitancy in EMS professionals wherein we selected a simple random sample of 19,062 nationally certified EMS professionals from the database.²² The voluntary, web-based survey invitation was sent via a unique link to EMS professionals’ provided e-mail addresses within the National Registry database. This unique link allowed for 1-to-1 matching with each respondent’s data in the National Registry database, preventing multiple attempts to be completed by the same individual or by multiple individuals with access to an invited participant’s e-mail. Our survey began with an informed consent form that explained the nature and risks of the study. The American Institutes for Research’s Institutional Review Board approved this study, which was deemed exempt from further review.

Study design

The objective of this cross-sectional study was to develop and validate a scale of COVID-19 vaccine concerns. We followed recommendations for writing items described by DeVellis²³ and the process outlined by Hinkin²⁴ for scale validation. For assessment of internal consistency reliability, construct validity, and criterion-related validity, the developed scale was evaluated against other previously validated scales that test similar vaccine hesitancy constructs (the Oxford COVID-19 Vaccine Hesitancy measure,¹⁶ and the Vaccine Hesitancy Scale,²⁵ described further below).

Procedure

Scale development

To develop a scale on COVID-19 vaccine concerns, we conducted a literature review using PubMed and Google Scholar. Specifically, we reviewed scales on general vaccine hesitancy,^25–28 and COVID-19 vaccine hesitancy and attitudes.^14,16,29 For content validity, we conducted interviews with 21 patrol officers, firefighters, and paramedics (the population who was of focus for the current survey validation) asking about their opinions and concerns around the COVID-19 vaccine to ensure our items adequately represented the full range of the construct. The most frequently mentioned concerns in these interviews included side effects, potential long-term effects of the vaccine, and potential for adverse safety issues due to the rapid development timeline. Due to the time sensitivity of the topic (i.e., needing to time the survey appropriately given the timing of vaccine releases), we were unable to release the survey in a pre-field test.

Data collection

Electronic questionnaires were sent to the study population in April 2021 following a tailored Dillman³⁰ method with reminders sent at one and two weeks after initial contact. Survey participation was voluntary and did not include an incentive to participate. Demographic data from the participants’ National EMS Certification database profile were linked to survey data and then deidentified for analysis.

Measures

COVID-19 vaccine concerns

The resulting scale was a 7-item measure on a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree). The scale was then assessed against the following previously developed and validated scales for evidence of validity.

COVID-19 vaccine hesitancy

COVID-19 vaccine hesitancy was measured with two items adapted from the Oxford COVID-19 Vaccine Hesitancy measure.¹⁶ Higher scores indicated more hesitancy: (a) “If my family or friends were thinking of getting a COVID-19 vaccination, I would: (1) Strongly encourage them, (2) Encourage them, (3) Not say anything to them about it, (4) Ask them to delay getting a vaccination, (5) Suggest they do not get a vaccination;” (b) “Taking a COVID-19 vaccination is: (1) Really important, (2) Important, (3) Neither important nor unimportant, (4) Unimportant, (5) Really unimportant.” Only two of the original seven items on the Oxford scale were relevant for inclusion due to the wording of the other five items being future-oriented (e.g., “If there is a COVID-19 vaccine available [I will want to get it as soon as possible/I will take it when offered/I’m not sure what I will do/I will put off (delay) getting it/I will refuse to get it/Don’t know]”), and the timing of the current study (which began months after widespread availability of the vaccines for our study population).

General vaccine hesitancy

General vaccine hesitancy was measured using the 9-item Vaccine Hesitancy Scale.²⁵ Items were on a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree). Higher scores indicated more hesitancy.

COVID-19 vaccination status

Vaccination status was determined by asking participants “Have you received a COVID-19 vaccine?” Response options were yes or no.

Demographics

Demographics included sex, age, race/ethnicity, urbanicity (residing in urban/suburban vs. rural), high-risk condition status, and education level. The nominal variable of sex was categorically designated as male or female. Age was a continuous variable. Race and ethnicity were dichotomized to non-minority (white, non-Hispanic) or minority (including Black or African American, Asian, Hispanic or Latino, Asian or Native Hawaiian or Pacific Islander), due to a small proportion of minority EMS professionals. Educational level was a categorical variable including less than high school/completed high school/obtained a General Education Development (GED) degree, some college, Associate’s degree, and Bachelor’s degree, or graduate degree.

Data analysis

Missing data were dropped listwise rather than imputed to avoid making assumptions concerning predicted responses while developing the scale. Descriptive statistics were evaluated for demographics. For Likert scales, items were reverse coded when necessary, and mean composites were computed. Cronbach’s alpha was used to compute internal consistency reliability. Pearson correlations were used to assess construct validity. Logistic regression was used to examine criterion-related validity, with model fit being assessed by the Hosmer–Lemeshow test. Area under the curve (AUC) analyses of receiver operating characteristic (ROC) curves were examined for the logistic regression models to determine prediction accuracy. To examine incremental validity of the scale in predicting COVID-19 vaccination status above and beyond any predictive validity of general vaccine hesitancy, we also tested a multivariable logistic regression model using both variables as predictors. The sample was split randomly to allow for an exploratory factor analysis (EFA) on one subsample and a confirmatory factor analysis (CFA) on the other subsample. For the EFA, maximum likelihood extraction with direct oblimin rotation was selected. For the CFA, maximum likelihood estimation was used. SPSS v.27³¹ was used for all analyses with the exception of CFA, for which LISREL v.10.3.3.26³² was used. Statistical tests were two-tailed, with p < .05 indicating statistical significance.

Results

Descriptive statistics

A total of 2,281 participants completed the COVID-19 Vaccine Concerns Scale (CVCS) (response rate = 12%) with the majority being male, white and non-Hispanic; their average age was 40. Participants’ education levels ranged from high school/GED to doctoral degrees. Participants lived in rural, suburban and urban areas. Twenty-eight percent of participants had a condition such as heart disease or obesity that put them at high risk for COVID-19. See Table 1.

Table 1. Demographic characteristics of participants.

Characteristic	n or Mean	% or SD
Gender
Male	1,495	65.5%
Female	757	33.2%
Missing	29	1.3%
Age	M = 40.09	SD = 13.4
Race & Ethnicity
White, Non-Hispanic	1,929	84.6%
All others	274	12.0%
Missing	78	3.4%
Educational Level
High school/GED	215	9.4%
Some College	610	26.7%
Associate’s	413	18.1%
Bachelor’s	508	22.3%
Master’s	142	6.2%
Doctorate	29	1.3%
Missing	364	16.0%
Urbanicity
Urban	423	18.5%
Suburban	924	40.5%
Rural	788	34.5%
Missing	146	6.4%
High risk condition^Citation1
Yes	641	28.1%
No	1,501	65.8%
Missing	139	6.1%

Abbreviations: GED = General Educational Development; M = Mean; SD = Standard deviation; 1 = cancer, chronic kidney disease, chronic obstructive pulmonary disease, heart disease, immunocompromised state, obesity, pregnancy, sickle cell disease, smoking, Type 2 diabetes mellitus.

Measure reliability and validity

The CVCS was evaluated for reliability with a Cronbach’s alpha of .89, suggesting high internal consistency reliability.³³ All items exhibited adequate inter-item correlation.

An EFA was conducted on a random subsample of 1,147 participants. The Kaiser–Meyer–Olkin value was .90 indicating sampling adequacy for factor analysis.³⁴ The determinant was .03, indicating lack of multicollinearity.³⁵ Only the first eigenvalue was above 1.0, with a value of 4.19 (explaining 59.86% of variance). The second eigenvalue was .68 (explaining 9.71% of variance). The scree plot (Figure 1) similarly suggested a one-factor solution. Factor loadings for all items were high (Table 2) and all were above the recommended cutoff point for inclusion of .30.³⁵

Figure 1. EFA scree plot for COVID-19 vaccine concerns scale.

Table 2. Factor loadings and inter-item correlations

Item	1	2	3	4	5	6	7
1. If a person has already had COVID-19 they do not need to get a vaccine	.66
2. I am worried I could get COVID-19 from a vaccine	.39**	.61
3. The risks of COVID-19 are less than the risks of a vaccine	.48**	.38**	.61
4. I am concerned about a COVID-19 vaccine causing severe adverse reactions (e.g., severe allergic reaction,death, etc.)	.49**	.50**	.52**	.78
5. I am concerned about the long-term side effects of getting a COVID-19 vaccine	.53**	.45**	.52**	.74**	.83
6. I am worried a COVID-19 vaccine could change my DNA	.50**	.54**	.47**	.59**	.64**	.80
7. COVID-19 and vaccinations are all part of a larger plot	.55**	.50**	.52**	.57**	.63**	.67**	.80

Note. EFA factor loadings on diagonal; numbers outside of diagonal are correlation coefficients; **p<.01.

A CFA was conducted on the remaining subsample of 1,134 participants. The overall model fit well, Χ2(14) = 189.26, p < .001; comparative fit index (CFI) = .95, root-mean-square error of approximation (RMSEA) = .11 (.09, .12), non-normed fit index (NNFI) = .93, standardized root mean square residual (SRMR) = .03. The CFI, NNFI, and SRMR values indicate good fit,³⁶ while the RMSEA value was out of bounds of the optimal criterion of .06.³⁶ All items loaded highly onto the latent factor (see Figure 2).

Figure 2. CFA results from COVID-19 vaccine concerns scale.

Construct validity

The adapted Oxford COVID-19 vaccine hesitancy scale¹⁶ and the general vaccine hesitancy scale²⁵ were reliable (α = .91 and .90, respectively).³³ The 7-item CVCS scale was highly and positively correlated with the adapted Oxford COVID-19 vaccine hesitancy scale¹⁶ (r = .82 ; p < .001; n = 2279) and the general vaccine hesitancy scale²⁵ (r = .71 , p < .001; n = 2144). These findings suggest high construct validity.

Criterion-Related validity

The mean score on the 7-item scale predicted COVID-19 vaccination status, B = −2.21, Exp(B) = .11 (95% CI = .09, .13) and was assessed for goodness of fit (non-significant Hosmer–Lemeshow test, 10 groups). The scale predicted a large amount of variance in vaccination status (Nagelkerke R² = .55) and AUC was high (AUC = .90), see Figure 3 for ROC. This indicates high criterion-related validity. Further, in a multivariable model with general vaccine hesitancy, the CVCS predicted COVID-19 vaccination status (B =  −2.07, Exp(B) = .13 [95% CI = .10, .16]) above and beyond general vaccine hesitancy (B =  −.40, Exp(B) = .67 [95% CI = .52, .87]). This multivariable model predicted a large amount of variance (Nagelkerke R² = .58), but only slightly more than the model with just the CVCS scale. The multivariable model had a high AUC (AUC = .91; see Figure 4 for ROC), but did not have good fit (significant Hosmer–Lemeshow test, 10 groups). Altogether, this indicates incremental validity of the CVCS scale above and beyond general vaccine hesitancy to predict COVID-19 vaccination status.

Figure 3. ROC curve for CVCS predicting COVID-19 vaccine status.

Figure 4. ROC curve for CVCS and general vaccine hesitancy predicting COVID-19 vaccine status.

Discussion

This study sought to develop and validate a scale to assess concerns around COVID-19 vaccines. The resultant CVCS scale is unifactorial and is composed of seven self-report items on a 5-point Likert scale. The scale showed high internal consistency reliability, construct validity, and criterion-related validity. Scores on the CVCS were correlated with scales of general and COVID-19 vaccine hesitancy, and scores significantly predicted COVID-19 vaccination status in a high-risk group of frontline healthcare professionals.

Our scale development was conducted at a time during which understanding of vaccine hesitancy was increasingly important to public safety, but few validated scales existed that were specifically tailored to address COVID-19 vaccine hesitancy. Many of the available scales had adapted existing surveys related to perspectives on general vaccine hesitancy or childhood vaccination.^17,37,38 Those scales that were developed specifically for COVID-19 vaccines have limitations, including wording that is outdated in light of the current availability of a vaccine (e.g., “If a COVID-19 vaccine was available at my local pharmacy, I would … ”),¹⁶ not considering specific COVID-19 vaccine concerns,¹⁹ only focusing on one aspect of concerns (adverse effects of the vaccine),²⁰ or being lengthy. Our survey adds to these resources by providing a succinct tool to assess topics specifically relevant to our evolving discovery of individuals’ perspectives that impact COVID-19 vaccine hesitancy, such as the role of existing immunity, new vaccine technology, and conspiracy theories. Additionally, because each item of our scale represents a unique, specific concern, considering responses to each item individually can enable better understanding of individuals’ specific concerns around COVID-19 vaccines. This information can help to provide insight about the particular issues that may be driving COVID-19 vaccine hesitancy for an individual or community, which, in turn, can inform targeted education and messaging to address these issues. Moreover, our scale shows evidence of criterion-related validity by predicting actual COVID-19 vaccination status, in contrast to other published scales that have examined associations with vaccine intentions rather than actual vaccination status.^{16,17,19–21}

Furthermore, while several general vaccine hesitancy scales have been correlated to willingness to accept vaccination (i.e., prior to the availability of COVID-19 vaccines),^16,39 few have yet been tested to predict self-reported vaccination status, now that vaccines have been available to all adults in the U.S. since April 2021.⁴⁰ A mean of the seven items on the CVCS can be used to predict both COVID-19 vaccine hesitancy and COVID-19 vaccination status. Our scale therefore offers a validated approach to understand COVID-19 vaccine hesitancy at a time in which the public is actively making the decision to receive the vaccine, or not. Addressing individuals’ concerns about COVID-19 vaccines, with messaging or education informed by efforts to understand vaccine hesitancy, could potentially increase COVID-19 vaccination rates. Increasing COVID-19 vaccinations will help efforts to reach herd immunity by reducing the prevalence of the virus and its transmission in the community, and can decrease the strain on overwhelmed healthcare systems by reducing the number of severely ill COVID-19 patients. It is critical that we continue to evaluate COVID-19 vaccine hesitancy as the pandemic evolves, especially as topics such as booster shots, FDA approval of vaccines, and vaccine mandates continue to shape perspectives about COVID-19 vaccination moving forward.

There are several limitations of this study. First, our sample was a random sample of EMS professionals, and there are possible limitations around generalizability to the general public. For example, racial/ethnic minorities, as well as females, are underrepresented in the EMS population, as compared the U.S. population.⁴¹ In our study, it was important to test the scale in a group of professionals at high risk for contracting COVID-19 to improve generalizability to other high-risk occupational groups, especially as vaccine hesitancy may place undue burden on these types of professionals. Regardless, the scale should be further tested in a general population to examine generalizability and measurement equivalence. Similarly, it would have been ideal to test the CFA in a separate sample, instead of creating random halves of one sample; however, this practice has been commonly used and shown to be adequate.²⁴ In addition, the RMSEA fit index for the CFA was above the commonly accepted criterion of .06;³⁶ nonetheless, the other three fit indices tested were indicative of good model fit. Further, due to the time sensitivity of the topic, we did not have the opportunity to field test the survey before deployment. Although this was a risk, we felt confident that the work we did to develop the scale before it was released was robust. Additionally, our response rate was fairly low, at 12%. While this is a typical response rate for surveys conducted in an EMS population,^42,43 it can lead to response and selection bias. Further, our measure of vaccine status was through self-report; future work should test its predictive validity with objective vaccine data. Finally, this scale was validated only in English; future work should seek to assess its validity in other languages.

Conclusions

The 7-item COVID-19 Vaccine Concerns scale showed high internal consistency reliability, construct validity, and criterion-related validity. We recommend a composite score of this scale be used to assess individuals’ level of concern, as a predictor of COVID-19 hesitancy and likeliness of COVID-19 vaccination. In addition, the individual items can be assessed to understand specific concerns to inform tailored messaging and education. As this scale was tested in the EMS population, future work should examine the validity of this scale in the general population.

Acknowledgments

The authors are grateful to the individuals who participated in this study, and to the National Registry of Emergency Medical Technicians that facilitated survey distribution.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the National Cancer Institute (NCI) [grant # CA260582], but the NCI was not involved in the data collection, analysis, nor the decision to present these findings.