Online study of health professionals about their vaccination attitudes and behavior in the COVID-19 era: addressing participation bias

ABSTRACT

Online surveys of health professionals have become increasingly popular during the COVID-19 crisis because of their ease, speed of implementation, and low cost. This article leverages an online survey of general practitioners’ (GPs’) attitudes toward the soon-to-be-available COVID-19 vaccines, implemented in October–November 2020 (before the COVID-19 vaccines were authorized in France), to study the evolution of the distribution of their demographic and professional characteristics and opinions about these vaccines, as the survey fieldwork progressed, as reminders were sent out to encourage them to participate. Focusing on the analysis of the potential determinants of COVID-19 vaccine acceptance, we also tested if factors related to survey participation biased the association estimates. Our results show that online surveys of health professionals may be subject to significant selection bias that can have a significant impact on estimates of the prevalence of some of these professionals’ behavioral, opinion, or attitude variables. Our results also highlight the effectiveness of reminder strategies in reaching hard-to-reach professionals and reducing these biases. Finally, they indicate that weighting for nonparticipation remains indispensable and that methods exist for testing (and correcting) selection biases.

KEYWORDS:

• Health care professionals
• online surveys
• selection bias
• attitudes
• practices
• covid-19
• weighting
• Heckman method

Introduction

Conducting surveys of health-care professionals’ (HCPs) perceptions, attitudes, and practices is hard, and their participation rates are often low. In the era of COVID-19, use of online surveys to reach these professionals has increased¹ because these provide a convenient, inexpensive, and safe method for collecting information from large numbers of participants in a short time. The range of secure software solutions available for online questionnaire surveys has facilitated their development in recent years. Online surveys of professionals, however, as of the general population, present the risk of significant participation bias.²

Two types of participation bias can be distinguished by whether they are related to observed (measured) or unobserved (unmeasured) variables. In the first case, weighting strategies that take these observed variables (e.g., age and gender) into account make it possible to obtain samples representative for these variables. Weighting is not systematically applied however, as this requires sampling frames that contain accurate information for the entire population of interest about certain essential individual characteristics, and these do not always exist. For example, although HCPs’ workload may be associated with their participation in surveys,³ this variable is rarely available. Bias can also arise if unobserved factors, such as their level of interest in public health, are linked to both survey participation and the dependent variables studied.

This article leverages a survey of general practitioners’ (GPs’) attitudes toward the soon-to-be-available COVID-19 vaccines, implemented in October–November 2020 (before the COVID-19 vaccines were authorized in France),⁴ to study the evolution of the distribution of their demographic and professional characteristics as the survey fieldwork progressed, as reminders were sent out to encourage them to participate, and simultaneously, of their opinions about these vaccines. Focusing on the analysis of the potential determinants of COVID-19 vaccine acceptance, we also tested if factors related to survey participation biased the association estimates.

Methods

Study design and population

We implemented an online cross-sectional survey within a representative national panel of 2,755 non-salaried GPs in France. The panel was set up in 2018, by random selection of GPs from an exhaustive database of health professionals (French national directory of health professionals).⁴ To be included in the panel, GPs had to be in private practice and not practice complementary and alternative medicine exclusively.

Data collection

We collected the data with a sequential mixed-mode design: participants were invited to take part online; if they had not completed the survey after five e-mail or text (mobile phone numbers were available for 79% of panel members) reminders over 4 weeks, at different times of day and on different days of the week, they were contacted by telephone. Participants were informed in the e-mail sending them the link to the questionnaire that the collected data and the results would be anonymous. They were informed that the questionnaire was about 15 minutes long, with two parts: one on their opinions and practices regarding vaccination in general, and another on the COVID-19 epidemic. The ethics board of the Conseil national de l’information statistique (France, CNIS, avis n°114/H030) approved the study protocol and questionnaire.

Questionnaire

The questionnaire asked participants two questions about these future COVID-19 vaccines: 1) the GPs’ willingness to be vaccinated themselves and 2) their willingness to recommend the vaccines to their patients. It used a five-point scale from “no, certainly not” and “no, probably not” to “yes, probably,” and “yes, certainly,” with a “don’t know” option. Scores ranged from 0 to 3. Besides queries about their uptake of last year’s seasonal influenza vaccine (Yes/No/Don’t remember), other questions probed perceptions of the safety of these urgently developed new vaccines and trust in the health authorities to ensure that vaccines are safe. This last item was asked to assess the extent to which GPs perceive that the French health authorities are putting in place effective mechanisms to monitor the safety of vaccines. The degree of institutional trust is a key factor in vaccine hesitancy.⁵ Finally, participants were asked about their perception of the medical severity of COVID-19 for the general population, on a scale from 0 (not severe) to 10 (extremely severe).

Workload

We created a categorical variable indicating GPs’ workload, based on their number of consultations and patient visits in 2017. These data were obtained from the French National Health Insurance Fund.

Sample size

A sample size of about 1000 respondents was required to estimate the percentage of GPs willing to accept these vaccines with a 95% confidence interval and an error margin of ± 3%.⁶

Statistical analysis

To analyze the evolution of the sample’s characteristics throughout the survey, we distinguished four periods by the reminders sent (Table 1) and compared the distribution of GPs’ characteristics and opinions with Chi-2 tests on unweighted data by period. We also weighted the sample for age, gender, region, workload, and density of GPs in the GPs’ practice area to compare weighted and unweighted percentages of these characteristics for the entire sample of respondents.

Table 1. Number of respondents to the survey per phase, French general practitioners, unweighted data, October to November 2020

Phase	N	%
No reminder^a (<09/10/2020)	256	21.17
1 to 2 reminders (09 to 18/10/2020)	411	34.00
3 to 5 reminders (19 to 25/10/2020)	184	15.22
Telephone interviews^b (>25/10/2020)	358	29.61
Total	1,209	100.00

^aReminders were made by sending an e-mail or a text message to the panel members who had not yet responded

^bGPs who had not completed the questionnaire online were contacted by trained professional interviewers (Kantar Public) to invite them to complete the questionnaire with them on the phone. During this phase, 74 GPs preferred completing the questionnaire online: these GPs have been put in the last category because, along with those who were interviewed by telephone, they were the last to participate.

We applied a previously published method to construct a score of presumptive acceptance of future COVID-19 vaccines,⁴ taking self-vaccination and recommendations to patients into account (alpha: 0.88, range [0–6]). Next, the score was transformed into a three-point variable named “COVID-19 vaccine acceptance”: high acceptance (score >4), moderate acceptance (score = 4), hesitancy or reluctance (score <4, or answered “don’t know” to at least one of the two items).

Finally, we focused on “Covid-19 vaccine acceptance” (polytomous dependent variable) to test whether potential differences between panel participants at inclusion and those participating in this survey might bias the results of a regression analysis studying factors associated with this dependent variable. We implemented a generalization of the Heckman selection model for polytomous outcomes that can test (and correct) for the presence of selection bias. It consists in a multinomial probit model with sample selection^7,8 (supplemental material S1). In these models, the first equation applied to the entire sample of panelists and analyzed factors (age, gender, region, workload, and GP density in the area of practice) potentially associated with both inclusion in the panel and participation in this survey (see supplemental material S2 for the specification of this equation). The other equation applied only to respondents to this survey (N = 1,209) and studied the factors associated with their COVID-19 vaccine acceptance. Such a model tests the correlation (rho) between the error terms of the two equations that may occur if any factors are associated with both survey participation and COVID-19 vaccine acceptance. A significant correlation means that estimates of the second equation are biased; the rho is then used to calculate unbiased estimates.

All analyses used two-sided p-values, defined statistical significance as p < .05, and were performed with Stata 14. The multinomial probit model was implemented using the cmp package.⁹

Results

In all, 1,209/2,755 GPs (43.9%) participated. The evolution of the number of respondents throughout the survey according to reminders is shown in Figure S2 in the supplemental material. Over the four survey periods, the proportions of GPs who were male, aged 60+, and had a high workload all increased significantly (Table 2). The proportion practicing complementary and alternative medicine (CAM) also varied significantly, but not linearly. The proportion trusting the Ministry of Health increased significantly throughout the survey, while that perceiving the pandemic as severe decreased significantly (Table 3). Nonetheless, neither the acceptability of COVID-19 vaccines nor reports of personal vaccination against seasonal influenza during the previous winter varied significantly by survey phase. The perception of the safety of future vaccines developed as an emergency response to an epidemic showed non-linear variations at the limit of significance (p = .07).

Table 2. Trends in respondents’ demographic and professional characteristics throughout the survey, French general practitioners, unweighted data, October to November 2020

	Survey phases of the survey
	No reminder	1–2 reminders	3–5 reminders	Telephone reminder &/or interview		Total
	N = 256	N = 411	N = 184	N = 358	p-value^a	N = 1,209
	Col. uW%	Col. uW%	Col. uW%	Col. uW%		uW%^b	W%^c
Gender
Male	49.22	43.07	52.72	64.80	< 0.001	52.27	60.87
Female	50.78	56.93	47.28	35.20		47.73	39.13
Age (years)
<50	51.95	52.80	40.22	36.31	< 0.001	45.82	31.18
50–59	28.13	22.38	26.63	26.82		25.56	32.24
60+	19.92	24.82	33.15	36.87		28.62	36.57
Region
National	63.28	58.64	66.30	62.29	0.20	61.87	85.68
Southeast	17.58	18.73	14.13	21.23		18.53	8.66
West	19.14	22.63	19.57	16.48		19.60	5.66
Workload
Min-Q1	30.47	33.09	25.00	23.74	0.003	28.54	23.33
Q1-Q3	55.47	52.31	54.35	52.23		53.27	50.79
Q3-Max	14.06	14.60	20.65	24.02		18.20	25.88
GPs density of practice area (deciles)
Decile 1	10.94	12.90	13.04	16.76	0.19	13.65	6.55
Deciles 2 to 10	89.06	87.10	86.96	83.24		86.35	93.45
Occasional practice of complementary and alternative medicine (CAM)
Yes	10.16	15.57	17.93	11.73	0.048	13.65	15.75
No/Don’t know	89.84	84.43	82.07	88.27		86.35	84.25

^ap-value: Chi-squared test

^buW: unweighted

^cW: weighted

Table 3. Trends of respondents’ opinions and attitudes throughout the survey, French general practitioners, unweighted data, October to November 2020

	Phases of the survey
	No reminder	1–2 reminders	3–5 reminders	Telephone reminder &/or interview		Total
	N = 256	N = 411	N = 184	N = 358	p-value^a	N = 1,209
	Col. %	Col. %	Col. %	Col. %		uW%^b	W%^c
Covid-19 vaccine acceptance
High acceptance	47.27	45.26	42.93	48.32	0.17	46.24	50.59
Moderate acceptance	23.83	27.01	27.72	30.73		27.54	24.39
Hesitancy or reluctance	28.91	27.74	29.35	20.95		26.22	25.02
I trust the ministry of health to ensure that vaccines are safe
Disagree/Don’t know^d	31.64	28.22	20.65	18.16	< 0.001	24.81	24.10
Agree	68.36	71.78	79.35	81.84		75.19	75.90
Were you vaccinated against seasonal influenza for the winter 2019–2020 season?
No/Don’t know^d	11.72	11.19	11.96	14.80	0.46	12.49	14.70
Yes	88.28	88.81	88.04	85.20		87.51	85.30
The safety of a vaccine developed in an emergency, during an epidemic, cannot be considered guaranteed
Disagree	44.53	38.44	37.50	46.37	0.07	41.94	42.45
Agree/Don’t know^d	55.47	61.56	62.50	53.63		58.06	57.55
In your opinion, for the population as a whole, how serious is COVID-19 on a scale of 0 to 10?
Moderate/High/Don’t know^d [5–10]	67.19	69.34	76.09	79.61	0.001	72.95	72.54
Low [0–4]	32.81	30.66	23.91	20.39		27.05	27.46

^ap-value: Chi-squared test

^buW: unweighted

^cW: weighted

^dDue to their small sizes, “don’t know” categories were merged with others.

Finally, in the Heckman model, the nonsignificant LR test of the correlation of the error terms of the two equations testing first survey participation and then vaccine acceptance determinants (p = .62, Table 4) indicated the absence of selection bias potentially due to panel participation and then attrition. The model showed that survey participation was lower among GPs who were older, worked in low GP-density areas or in southeastern France, had high workloads, or practiced CAM occasionally; participation was higher among women, however. Compared to GPs with high acceptance of COVID-19 vaccines, hesitancy was higher among women, doctors doubtful about the safety of rapidly developed vaccines, perceiving the epidemic’s medical severity as low, distrustful of the health authorities, and those not vaccinated against influenza in the winter of 2019–20. The prevalence of moderate acceptance was lower among older GPs, those with a higher workload and who were vaccinated against seasonal influenza in 2019–20. It was, however, higher among GPs who occasionally practiced CAM, who thought the safety of vaccines developed during an emergency could not be considered guaranteed and who considered the medical severity of COVID-19 to be low.

Table 4. Factors associated with Covid-19 vaccines acceptance among French GPs, polytomous probit regressions with sample selection (on all GPs eligible for the panel, N = 11,146), unweighted data, October to November 2020

			Covid-19 vaccine
			acceptance (ref. High^a)
	Participation in the study^b (ref. No)		Moderate		Hesitancy/reluctance
	N = 11,146		N = 1,209		N = 1,209
	Coef^c	[CI 95%]^d	Coef	[CI 95%]	Coef	[CI 95%]
Characteristics
Gender (ref. Male)
Female	0.11**	[0.04,0.18]	0.18	[−0.07,0.44]	0.48**	[0.16,0.80]
Age (ref. < 50)
50 to 60	−0.31***	[−0.39,-0.24]	−0.39*	[−0.71,-0.08]	−0.16	[−0.56,0.24]
> 60	−0.38***	[−0.46,-0.30]	−0.54**	[−0.86,-0.21]	−0.24	[−0.68,0.21]
Region (ref. National)
Southeast	−0.19***	[−0.27,-0.10]	0.05	[−0.25,0.34]	0.38*	[0.06,0.70]
West	−0.10*	[−0.19,-0.02]	−0.07	[−0.35,0.20]	−0.19	[−0.53,0.15]
Low density practice area (ref. No)
Yes	−0.11**	[−0.19,-0.03]
Workload (ref. Low [Q1])
Moderate [Q2-Q3]	−0.04	[−0.12,0.03]	−0.17	[−0.42,0.07]	−0.25	[−0.54,0.03]
High [Q4]	−0.19***	[−0.28,-0.09]	−0.54**	[−0.87,-0.22]	−0.24	[−0.63,0.16]
Occasional practice of complementary and alternative medicine (CAM) (ref. No/Don’t know)
Yes			0.36*	[0.04,0.68]	0.22	[−0.14,0.59]
Perceived risks of new vaccines in emergencies and of Covid-19
The safety of a vaccine developed in an emergency, during an epidemic, cannot be considered guaranteed (ref. Disagree)
Agree			0.97***	[0.67,1.28]	1.93***	[1.54,2.31]
Don’t know			0.56**	[0.21,0.91]	1.31***	[0.87,1.76]
Perceived risks of Covid 19
In your opinion, for the population as a whole, how serious is COVID-19 on a scale of 0 to 10?
Low [0–4]			0.26*	[0.03,0.50]	0.47***	[0.20,0.74]
Trust in science and in the ministry of health
I trust the ministry of health to ensure that vaccines are safe (ref. Disagree)
Agree			−0.31	[−0.64,0.01]	−0.86***	[−1.22,-0.50]
Don’t know/no response			−0.11	[−0.61,0.38]	0.10	[−0.40,0.61]
Personal vaccination
Were you vaccinated against seasonal influenza for the winter 2019–2020 season? (ref. No/non-response)
Yes			−0.43*	[−0.79,-0.07]	−0.87***	[−1.26,-0.48]

P-values: *: <0.05; **: <0.01; ***: <0.001; ^aBecause the dependent variable (i.e. acceptance) has three categories ((i) ‘high acceptance’, (ii) ‘moderate acceptance’ and (iii) ‘hesitancy/reluctance’), there are two comparisons (multiple polytomous probit regression): ‘moderate acceptance’ is compared to ‘high acceptance’ (reference), and ‘hesitancy/reluctance’ is also compared to ‘high acceptance’. Positive probit coefficient indicate a positive association with the dependent variable (e.g. female were more likely to participate in the study) whereas negative coefficients indicate a negative association (e.g. older GPs were less likely to participate in the study). The higher the absolute value of the probit coefficient, the greater the strength of the association.

^bFactors associated with participation to the study; these would have been used to correct the estimates of the next two columns, in the presence of selection bias; in the present analysis however, we did not find evidence for such bias.

^cProbit coefficient; ^d Confidence Interval; p-value LR test: 0.62 (comparison between models with and without sample selection; this non-significant result indicates there is no evidence for selection bias); rho_moderate = 0.49 (correlation between study participation and moderate acceptance); rho_hesitancy = −0.15 (correlation between study participation and hesitancy)

Discussion

In this online survey of GPs, respondents differed significantly from nonrespondents for several characteristics (notably, age and workload) (Table 4). Among respondents, rapid responders also differed significantly from hard-to-reach GPs (who responded after multiple reminders) for gender, and again age and workload.

Important selection processes are therefore at work in surveys of HCPs. The individual characteristics underlying these biases are likely to be associated with various behaviors, opinions, and attitudes toward patient care. For example, female GPs devote more time to prevention and screening activities than male GPs;¹⁰ younger doctors are aware of and adhere to good practice guidelines more often than older ones;¹¹ and GPs with higher workloads tend to prescribe medication more often and spend less time with their patients than doctors with lower workloads.¹² Specifically concerning vaccination, our previous work showed that GPs younger than 50 y, female, or with the highest workloads tend to recommend the vaccines on the official schedule more frequently to patients than the others.³ In contrast, female HCPs (GPs and nurses) and those aged younger than 40 y expressed hesitation about upcoming COVID-19 vaccines more frequently.^4,13,14 The prevalence of vaccine hesitancy among GPs also varies considerably between regions in the same country, independently of demographic and professional characteristics.¹⁵ During the 2009 A/H1N1 pandemic, GPs with the highest workloads and those in group practices were willing to accept the A/H1N1 pandemic vaccines more often than other GPs, while we observed no differences by age or gender.¹⁶ These results among French GPs show that the direction of the observed links between their characteristics and their vaccination attitudes and behaviors may vary by vaccine type and event.

Behaviors and attitudes of CAM doctors toward patient management differ from those of other doctors: concerning vaccination, they are much more frequently hesitant and much less likely to recommend vaccines on the official calendar than non-CAM GPs.^3,17 The significant variations in the percentage of CAM GPs between the survey phases did not follow a linear trend, perhaps due to the relatively small fraction of the sample they accounted for (N = 190).

The variation over the course of the survey in GPs’ perceptions of the pandemic risks and confidence in the health authorities is another important result: as potential determinants of various attitudes and behaviors of GPs,¹⁸ these variables are key indicators to monitor throughout a health crisis – such as COVID-19 – to follow GPs’ adherence to the recommended management. The variation in the perception of the future vaccines’ safety over the course of the survey (Table 3) was near the border of significance. It too is a crucial indicator to be monitored and may be sensitive to events, such as scientific publications about the efficacy and safety of new vaccines, marketing authorizations, and recommendations by professional societies.

An important methodological takeaway is the importance in online surveys of HCPs of applying a reminder strategy to improve the representativeness of survey participants.² Our experience shows that it is appropriate and feasible to go up to 5 reminders, without unduly burdening the survey costs. Recourse to telephone reminders and interviews at the end of a survey could be an optimal solution in terms of participation as it makes it possible to contact and include hard-to-reach GPs. But this must be balanced against cost and feasibility considerations. In addition, the potential differential reporting bias between collection by an interviewer and direct collection without an intermediary in an online questionnaire should also be taken into account.^19,20

In addition to a reminder strategy, our results also indicate the need to weight the data, especially when participation rates are relatively modest (here, 43.9%). Comparison of the unweighted and weighted results of our study showed non-negligible differences in age, gender, and volume of activity. Nonetheless, it is reassuring that the differences for the variables of interest studied here were more modest (4 to 5 percentage points for acceptance of COVID-19 vaccines, Table 3). As surveys among HCPs are likely to be repeated in a pandemic to monitor various aspects of their attitudes and behaviors as we did in France in the frame of our panel,²¹ weighting remains essential and ideally should include workload, provided that this information is available in both national databases and surveys.

Our results are also reassuring regarding potential selection effects due to participation in a survey since Heckman’s method did not suggest that they played a role in the analysis of factors associated with the acceptance of COVID-19 vaccines.

The results of this study must be interpreted with caution however. First, they relate to non-salaried GPs while the demographic and professional characteristics associated with being hard-to-reach in a survey may vary according to the type of health profession, practice setting (community, hospital), and country. However, workload is a characteristic that can reasonably be assumed to be common to all these professions. Furthermore, the impact of selection bias is likely to vary according to the dependent variable.

In any case, this type of bias should be systematically tested: to the extent possible, collecting the necessary data should be planned before survey implementation. Moreover, recent methodological developments make it possible to take unobserved factors into account in data weighting.²²

Disclosure of potential conflicts of interest

None to declare.

Acknowledgments

We thank all the participants of this survey, Jo Ann Cahn for supervising our English, and Maxime Bergeat for his invaluable advice.

Supplementary material

Supplemental data for this article can be accessed on the publisher’s website at https://doi.org/10.1080/21645515.2021.1921523

Additional information

Funding

This study was supported by grants from the Direction de la Recherche, des Etudes, de l’Evaluation et des Statistiques (DREES)/Ministère des solidarités et de la santé (grant 2102173353) and from the Agence Nationale de la Recherche (ANR) in the frame of the call for projects 2020 “Recherche-Action Covid-19”. [2102173353]