Confounding and Negative Control Methods in Observational Study of SARS-CoV-2 Vaccine Effectiveness: A Nationwide, Population-Based Danish Health Registry Study

Abstract

Background

Observational studies of SARS-CoV-2 vaccine effectiveness are prone to confounding, which can be illustrated using negative control methods.

Methods

Nationwide population-based cohort study including two cohorts of Danish residents 60–90 years of age matched 1:1 on age and sex: A vaccinated and a non-vaccinated cohort, including 61052 SARS-CoV-2 vaccinated individuals between 1 March and 1 July 2021 and 61052 individuals not vaccinated preceding 1 July 2021. From these two cohorts, we constructed negative control cohorts of individuals diagnosed with SARS-CoV-2 infection or acute myocardial infarction, stroke, cancer, low energy fracture, or head-trauma. Outcomes were SARS-CoV-2 infection, negative control outcomes (eg, mammography, prostate biopsy, operation for cataract, malignant melanoma, examination of eye and ear), and death. We used Cox regression to calculate adjusted incidence and mortality rate ratios (aIRR and aMRR).

Results

Risks of SARS-CoV2 infection and all negative control outcomes were elevated in the vaccinated population, ranging from an aIRR of 1.15 (95% CI: 1.09–1.21) for eye examinations to 3.05 (95% CI: 2.24–4.14) for malignant melanoma. Conversely, the risk of death in the SARS-CoV-2 infected cohort and in all negative control cohorts was lower in vaccinated individuals, ranging from an aMRR of 0.23 (95% CI: 0.19–0.26) after SARS-CoV-2 infection to 0.50 (95% CI: 0.37–0.67) after stroke.

Conclusion

Our findings indicate that observational studies of SARS-CoV-2 vaccine effectiveness may be subject to substantial confounding. Therefore, randomized trials are essential to establish vaccine efficacy after the emergence of new SARS-CoV-2 variants and the rollout of multiple booster vaccines.

Plain Language Summary

Why was this study done: After the emergence of new SARS-CoV-2 variants and the rollout of multiple booster SARS-CoV-2 vaccines, the impact of vaccination on risk of SARS-CoV-2 infection and death after the infection has mainly been explored in observational studies. We used negative control methods to investigate whether confounding affects the results of observational SARS-CoV-2 vaccine effectiveness studies.

Findings: We used Danish registry data obtained during the SARS-CoV-2 vaccine roll-out to conduct a nationwide, matched population-based cohort study of Danish residents 60–90 years in which we compared vaccinated individuals with non-vaccinated individuals. Compared with unvaccinated individuals, vaccinated individuals had increased risks of SARS-CoV2 infection but also had increased risks of all negative control outcomes (mammography, prostate biopsy, operation for cataract, malignant melanoma, examination of eye and ear). The risk of death after SARS-CoV2 infection was lower in the vaccinated cohort, as was the risk of death after acute myocardial infarction, stroke, cancer, low energy fracture, and head-trauma.

Meaning: The negative control methods indicate that observational studies of SARS-CoV-2 vaccine effectiveness may be prone to substantial confounding which may impact the observed associations. This bias may both lead to underestimation of vaccine effectiveness (increased risk of SARS-CoV2 infection among vaccinated individuals) and overestimation of the vaccine effectiveness (decreased risk of death after of SARS-CoV2 infection among vaccinated individuals). Our results highlight the need for randomized vaccine efficacy studies after the emergence of new SARS-CoV-2 variants and the rollout of multiple booster vaccines.

Keywords:

• COVID-19
• vaccines
• cohort studies
• bias
• confounding factors
• epidemiologic

Introduction

Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) causes coronavirus disease 2019 (COVID-19). In the initial phase of the SARS-CoV-2 pandemic, randomized trials demonstrated that SARS-CoV-2 vaccines decreased the risks of SARS-CoV-2 infection and hospitalization due to COVID-19.^1,2 Since the emergence of the Omicron variant of SARS-CoV-2 in late 2021, the effectiveness of the vaccines against infection has decreased.^3–5 Because the evidence supporting licensing of new bivalent-variant-updated mRNA SARS-CoV-2 vaccines was based on immunogenicity studies rather than trials, observational studies are necessary to assess population effectiveness.^6,7

SARS-CoV-2 vaccines are assumed to have two potential effects: 1) decreasing the risk of SARS-CoV-2 infection and 2) improving prognosis (eg, survival) after infection.⁸ Thus, vaccine effectiveness studies aim to examine risk and prognosis in routine clinical care.

SARS-CoV-2 vaccine effectiveness has been evaluated in few post-licensing observational studies. Observational study attempting to assess vaccine effectiveness must address complex confounding issues.^9,10 Confounding has been demonstrated in studies of influenza vaccine effectiveness and has been proposed to influence observational SARS-CoV-2 vaccine effectiveness studies.^9–16 Main confounding in observational studies of vaccine effectiveness stems from confounding by indication, healthy vaccine bias and confounding by health-seeking behaviour.⁹ Because most observational studies of vaccine effectiveness have used administrative healthcare databases, which often lack data on important confounders, such as lifestyle, behaviour and socioeconomic status, estimates might be affected by residual and unmeasured confounding.¹⁴

Negative control methods have been proposed as an approach to assessing the effects of such biases.¹⁷ A negative control outcome is one that cannot plausibly be caused by the exposure of interest yet shares the same confounding structure as that of the vaccine with respect to association with the primary outcome.^18,19 Herein, we used negative control methods to assess the likelihood of confounding in assessment of the effects of SARS-CoV-2 vaccination on infection. To assess the likelihood of confounding in assessment of the effects of vaccination on mortality among people with SARS-CoV-2 infection, we examined the association between vaccination and death in five different populations, each diagnosed with a condition in which survival is assumed not to be affected by SARS-CoV-2 vaccination. The presence of an association between the SARS-CoV-2 vaccination and negative control outcomes or between SARS-CoV-2 vaccination and death in the negative control cohorts would constitute evidence of confounding in the association between the primary exposure and primary outcome.

Materials and Methods

We used Danish health-registry and administrative data to conduct nationwide population-based cohort analyses. In all analyses, exposure was having received first SARS-CoV-2 vaccination between 1 March and 1 July of 2021 vs not having received SARS-CoV-2 vaccine 1 July 2021. First, we performed a risk study, in which the outcomes were times to the first positive SARS-CoV-2 test and eight negative control outcomes. Second, we assessed whether SARS-CoV-2 vaccination was associated with death among people who tested positive for SARS-CoV-2 infection and among people diagnosed with five non-infectious diseases with high short-term mortality (negative control cohorts).

Settings

Denmark has a free tax-funded healthcare system.²⁰ The first patient infected with SARS-CoV-2 in Denmark was diagnosed on 27 February 2020, and the first SARS-CoV-2 vaccine was administered on 27 December 2020. Target groups for vaccinations received individual invitations distributed through the secure governmental e-mail system. The rollout started with high-risk groups, defined as people of advanced age, people at high risk of severe COVID-19 and frontline healthcare workers. The Danish tax-funded healthcare system provided vaccination, testing and treatment for SARS-CoV-2 free of charge to all Danish residents.²¹

Data Sources

Data from Danish registries were collected as previously described.^22,23 The unique personal identification number assigned to each resident at birth or immigration into Denmark was used to merge the data from several registers.²⁴ In brief, data on sex, date of birth, immigration, emigration, and death were obtained from the Danish Civil Registration System.²⁵ Data on SARS-CoV-2 polymerase chain reaction (PCR) and antigen tests were extracted from the national COVID-19 surveillance system maintained by the Statens Serum Institut, which captures all results of SARS-CoV-2 tests performed in public and private testing facilities (including hospitals).²⁶ SARS-CoV-2 vaccination status was ascertained from the Danish Vaccination Register, which contains individual-level information on the dates and types of vaccines administered in Denmark.²⁷ Because reporting of vaccinations to this register is mandatory, the vaccination status information is considered nearly complete. Data on discharge diagnoses and procedures were extracted from the Danish National Patient Registry, which includes coded diagnoses, on the basis of International Classification of Diseases, Tenth Revision (ICD10) codes, from all Danish hospitals.²⁸

Study Cohorts

We identified all Danish residents who were alive and registered as living in Denmark on 1 July 2021 (Supplementary Figure 1). Because the mortality is lower in younger individuals, we limited our study population to individuals born in Denmark between 1 July 1931 and 1 July 1961 (60–90 years of age on 1 July 2021).

From this population, we constructed two cohorts:

Non-vaccinated cohort: All individuals, who had not received a first SARS-CoV-2 vaccination by 1 July 2021 (non-exposed). These individuals could have been vaccinated after 1 July 2021.

Vaccinated cohort: All individuals who received a first SARS-CoV-2 vaccine between 1 March and 1 July 2021 (exposed) matched at random 1:1 by sex and year of birth to the non-vaccinated cohort.

The two cohorts were used for the risk analyses, in which the date of study inclusion was 1 July 2021.

To estimate the association between SARS-CoV-2 vaccination and death, from the two source cohorts described above, we extracted the populations of individuals diagnosed after 1 July 2021 with SARS-CoV-2 infection (positive SARS-CoV-2 test) and five negative control cohorts, including individuals diagnosed after 1 July 2021 with acute myocardial infarction, stroke, cancer, low energy fracture, or head trauma (Supplementary Appendix 1). In these six cohorts, study entry was the date of diagnosis of the respective disease. Of note, this approach can potentially induce a second source of bias, collider stratification bias and therefore is not an ideal method to assess vaccine effectiveness against death. However, because this approach is commonly used in the literature to assess vaccine effectiveness against mortality and severe illness, and our goal was to demonstrate the likelihood of residual confounding in such analyses, we ignored that source of bias and focused only on the uncontrolled confounding.^29,30

Exposure and Outcomes

In all analyses, the exposure was a first SARS-CoV-2 vaccination between 1 March and 1 July of 2021. In the risk analyses, the outcomes were time to the first positive SARS-CoV-2 test, mammography screening, prostate biopsy, cataract surgery, examination due to a positive faecal blood screening result, non-melanoma skin cancer, malignant melanoma, and in-hospital examination of the eye or ear (Supplementary Appendix 1). We assumed that SARS-CoV-2 testing frequency and thereby the risk of positive test results were associated with health-seeking behaviour. Therefore, we used negative control outcomes expected to have similar health-seeking confounding structures as vaccination. In the mortality analyses, the outcome was 90-day all-cause mortality. The specified conditions are described in the Appendix, and the date of death was extracted from the Danish Civil Registration System.

Covariates

To decrease measured confounding to the greatest extent possible, we controlled for the following variables considered confounders of the effects of SARS-CoV-2 vaccination: age at study inclusion, sex; morbidity, calculated with the Charlson Comorbidity Index (grouped as scores of 0, 1, 2, 3, 4, 5 and >6);³¹ number of hospital contacts and number of inpatient hospitalization days in the year before study inclusion (grouped as 0, 1–5, 6–10 and >11 days); and geographical region (98 Danish municipalities categorized in quartiles according to the mean negative control outcomes in the municipality). All potential confounders were calculated according to the status on 1 July 2021.

Statistical Analyses

For analysis of effects of SARS-CoV-2 vaccination on the risk of SARS-CoV-2 infection or a negative control outcome (risk studies), person-time accrued from the study inclusion (1 July 2021) until diagnosis with the relevant outcome, death, emigration, or censoring on 1 July 2023, whichever occurred first. For analysis of the effect of vaccination on mortality (prognosis studies), person-time accrued from the date of diagnosis of SARS-CoV-2 infection or of one of the five diseases until all-cause death, emigration, or censoring on 1 July 2023 or at day 90, whichever occurred first.

We computed Kaplan–Meier life tables for time-to-event data and used Cox regression to calculate incidence rate ratios (IRR) and mortality rate ratios (MRR) with corresponding 95% confidence intervals. We calculated unadjusted IRRs and MRRs and adjusted for the covariates described above.

Because the exposure (being vaccinated or not vaccinated against SARS-CoV-2 before July 2021) was fixed for the total study period, the vaccination status of individuals in the non-vaccinated population could potentially shift during the study period. We therefore repeated the analyses and censored the time at the date of the first vaccination in the non-vaccinated cohort. Furthermore, because some individuals in the negative control populations might possibly have died from COVID-19, we repeated the prognosis analyses but censored person-time at the first SARS-CoV-2 test. Finally, we repeated the risk analyses with death as a competing risk.

Results

We included 61046 non-vaccinated individuals and 61046 SARS-CoV-2 vaccinated individuals in the primary risk analysis of the effect of vaccination on infection. The cohort included slightly more women (52%) than men, and almost 40% were registered with morbidity (Charlson Comorbidity Index score >0). The median age at the time of study inclusion was 69.0 years (Table 1). The use of healthcare services was greater in the vaccinated than non-vaccinated cohort. Vaccinated individuals had higher frequencies of SARS-CoV-2 testing and all negative control outcomes before study inclusion than non-vaccinated individuals. These findings suggested a potential for surveillance bias because vaccinated people might have undergone more testing, or differences in behaviour might have occurred if vaccinated people were more likely than non-vaccinated people to socially mix with others. At the end of the study, 19190 (31.4%) of the non-vaccinated population had been vaccinated with SARS-CoV-2 vaccine for the first time.

Table 1 Characteristics of the Study Cohorts

Characteristic	Vaccinated	Non-Vaccinated
N	61,046	61,046
Female	31,374 (51.4%)	31,374 (51.4%)
Age (IQR)	69.0 (63.8–76.0)	69.0 (63.8–76.0)
Observation time (years)	119,432	112,765
Charlson=0	36,936 (60.5%)	38,260 (62.7%)
Charlson=1	9,870 (16.2%)	9,796 (16%)
Charlson=2	8,008 (13.1%)	6,691 (11%)
Charlson=3	3,194 (5.2%)	3,036 (5%)
Charlson=4	1,334 (2.2%)	1,388 (2.3%)
Charlson=5	618 (1%)	723 (1.2%)
Charlson=6	1,086 (1.8%)	1,152 (1.9%)
Number of SARS-CoV-2 tests (IQR)	0 (1–5)	1 (4–9)
Positive SARS-CoV-2 tests	1,537 (2.5%)	2,379 (3.9%)
Vaccinated as of 1 July 2023	61,046 (100%)	19,190 (31.4%)
Outpatient visits (IQR)	0 (0–2)	0 (1–2)
Geographical region quartile 1	13,462 (22.1%)	14,564 (23.9%)
Geographical region quartile 2	15,884 (26%)	14,998 (24.6%)
Geographical region quartile 3	19,179 (31.4%)	20,352 (33.3%)
Geographical region quartile 4	12,517 (20.5%)	10,509 (17.2%)
Mammography screening	25,542 (81.4%)	16,667 (53.1%)
Prostate biopsy	2,327 (7.8%)	1,293 (4.4%)
Examination associated with screening for blood in faeces	4,348 (7.1%)	1,956 (3.2%)
Cataract operation	9,003 (14.7%)	6,799 (11.1%)
Melanoma	727 (1.2%)	346 (0.6%)
Skin cancer	2,147 (3.5%)	1,362 (2.2%)
In hospital examination of the eye	9,704 (15.9%)	7,870 (12.9%)
In hospital examination of the ear	9,537 (15.6%)	6,926 (11.3%)
Acute myocardial infarction	2,249 (3.7%)	2,331 (3.8%)
Stroke	3,046 (5%)	3,810 (6.2%)
Cancer	7,651 (12.5%)	5,925 (9.7%)
Trauma	32,728 (53.6%)	30,552 (50%)
Low energy fracture	7,862 (12.9%)	8,323 (13.6%)
Head trauma	8,247 (13.5%)	9,287 (15.2%)

We observed a 25% adjusted increased rate of diagnosis of SARS-CoV-2 in vaccinated compared to non-vaccinated individuals (1.25; 95% confidence interval 1.22 to 1.28). If unbiased, this finding would suggest that vaccination increased the risk of infection by approximately 25%. The risk of all negative control outcomes was also greater in the vaccinated than non-vaccinated cohort, ranging from 1.15 (95% confidence interval 1.09 to 1.21) for eye examinations to 3.05 (95% confidence interval 2.24 to 4.14) for malignant melanoma. As these outcomes are not expected to be causally associated with vaccination, these findings suggest confounding of the vaccination-infection estimate (Table 2 and Figure 1). Including death as competing risk only changed the risk estimates marginally (Table 2).

Table 2 Risk of Positive SARS-CoV-2 Tests and Negative Control Outcomes in Individuals Who Had and Had Not Received First SARS-CoV-2 Vaccine as of 1 July 2021

Outcome	Number with Outcome		Vaccinated vs Non-Vaccinated		Vaccinated vs Non-Vaccinated*
	Vaccinated	Non-Vaccinated	IRR Unadjusted	IRR Unadjusted^§	IRR Unadjusted	IRR Adjusted^§
First positive SARS-CoV-2 test	20,180	14,785	1.32 (1.29–1.35)	1.32 (1.29–1.35)	1.38 (1.35–1.41)	1.29 (1.27–1.32)
Mammography screening	11,755	5231	2.43 (2.35–2.51)	2.43 (2.35–2.51)	2.56 (2.48–2.64)	2.63 (2.54–2.72)
Prostate biopsy	338	178	1.84 (1.54–2.21)	1.84 (1.54–2.21)	1.97 (1.64–2.36)	1.91 (1.59–2.30)
Examination due to a positive faecal blood screening result	1,171	530	2.10 (1.89–2.32)	2.10 (1.89–2.32)	2.22 (2.00–2.46)	2.10 (1.89–2.33)
Operation for cataract	1,804	1,160	1.53 (1.42–1.65)	1.53 (1.42–1.65)	1.63 (1.51–1.75)	1.58 (1.46–1.71)
Skin cancer	510	300	1.63 (1.41–1.88)	1.63 (1.41–1.88)	1.72 (1.49–1.99)	1.63 (1.41–1.88)
Malignant melanoma	186	54	3.27 (2.42–4.43)	3.27 (2.42–4.43)	3.46 (2.56–4.69)	3.24 (2.37–4.42)
Examination of eye	3,716	2,630	1.36 (1.29–1.43)	1.36 (1.29–1.43)	1.42 (1.35–1.50)	1.25 (1.18–1.31)
Examination of ear	3,551	2,365	1.44 (1.37–1.52)	1.44 (1.37–1.52)	1.51 (1.43–1.59)	1.39 (1.32–1.47)

Notes: *With death as competing risk. ^§Adjusted for age, sex, Charlson Comorbidity Index, number of inpatient and outpatient hospital contact in the year before study inclusion and geographical region.

Abbreviation: IRR, incidence rate ratio.

Figure 1 Cumulative incidence (%) of first positive SARS-COV-2 test and negative control outcomes among non-vaccinated and vaccinated individuals.

In contrast, the risk of death after SARS-CoV-2 infection was substantially lower in the vaccinated cohort (0.23; 95% confidence interval 0.19 to 0.26). If the findings were unbiased, these findings would suggest that vaccination dramatically decreased the risk of mortality after infection. However, vaccination was also associated with decreased mortality in all negative control cohorts, ranging from 0.36 (95% confidence interval 0.27 to 0.48) for trauma to 0.50 (95% confidence interval 0.37 to 0.67) for stroke (Table 3 and Figure 2), although death after any of these outcomes is very unlikely to be affected by vaccination. These estimates changed only marginally after adjustment for available confounders (Tables 2 and 3) and censoring at the date of the first SARS-CoV-2 test and the date of the first SARS-CoV-2 vaccination in the non-vaccinated cohort (Table 3). The findings also suggest that the protective effect of vaccination against death is subject to downward confounding.

Table 3 Risk of Death After SARS-CoV-2 Infection and in Negative Control Population in Individuals Who Have and Have Not Received First SARRAS-CoV-2 Vaccine 1 July 2021

Cohorts	Number in Cohort		Number of Deaths		Vaccinated vs Non-Vaccinated		Vaccinated vs Non-Vaccinated*
	Vaccinated	Non-Vaccinated	Vaccinated	Non-Vaccinated	MRR_unadjusted	MRR_adjusted^§	MRR_unadjusted	MRR_adjusted^§
Positive SARS-CoV-2 test	20,180	14,789	211	702	0.22 (0.19–0.25)	0.23 (0.19–0.26)	NA	NA
Acute myocardial infarction	376	383	40	65	0.58 (0.38–0.90)	0.48 (0.31–0.75)	0.64 (0.38–1.07)	0.53 (0.31–0.90)
Stroke	910	809	72	140	0.56 (0.42–0.75)	0.50 (0.37–0.67)	0.63 (0.45–0.86)	0.53 (0.38–0.74)
Diagnosis of cancer	1,622	1,971	208	337	0.48 (0.41–0.57)	0.46 (0.38–0.54)	0.46 (0.38–0.57)	0.44 (0.36–0.53)
Trauma	2,225	2,417	69	169	0.37 (0.28–0.49)	0.36 (0.27–0.48)	0.37 (0.27–0.51)	0.33 (0.24–0.46)
Low energy fracture	1,451	1,473	60	133	0.44 (0.32–0.60)	0.43 (0.32–0.59)	0.45 (0.32–0.64)	0.43 (0.30–0.62)
Head trauma	1,028	1,072	40	74	0.53 (0.36–0.78)	0.47 (0.32–0.70)	0.50 (0.32–0.77)	0.42 (0.27–0.66)

Notes: *Time censored at positive SARS-CoV-2 test or SARS-CoV-2 vaccination (in the non-vaccinated cohort). ^§Adjusted for age, sex, and Charlson Comorbidity Index.

Abbreviations: NA, not applicable; MRR, mortality rate ratio.

Figure 2 Cumulative incidence (%) of death after positive SARS-COV-2 test and after diagnosis of non-infectious diseases among non-vaccinated and vaccinated individuals.

Discussion

In this nationwide population-based cohort study, the risks of SARS-CoV-2 infection and negative control outcomes were greater in people vaccinated before 1 July 2021 than non-vaccinated. Furthermore, the risk of death was substantially lower in vaccinated than non-vaccinated people after SARS-CoV-2 infection and after the diagnosis of common non-infectious diseases with high short-term mortality. Our results suggest substantial confounding of the effects of the vaccine on infection and death despite adjustment for measured confounding.

Importantly, our study was intended to assess only the presence of confounding in observational settings. We cannot draw conclusions regarding vaccine effectiveness based on these analyses.

We did not compare time vaccinated vs time not vaccinated; instead, we compared individuals vaccinated before 1 July 2021 with individuals either vaccinated later or not vaccinated. However, most individuals in the non-vaccinated cohort remained non-vaccinated during the entire follow-up. The exposure was therefore a combination of the effect of SARS-CoV-2 vaccination and the incentive to follow recommendations from healthcare authorities.

Individuals with serious comorbidities were the first people invited to receive SARS-CoV-2 vaccination, thus leading to confounding by indication. To decrease confounding by indication, we did not include individuals vaccinated early in the vaccine roll-out and yet we still found substantial residual confounding.

Our results support the hypothesis that it is difficult to remove confounding by health-seeking behaviour in observational studies of interventions. The two cohorts were well matched with respect to age and sex, and importantly, the cohorts did not considerably differ with respect to Charlson Comorbidity Index Score or geographical distribution. Despite these similarities, the non-vaccinated cohort, compared with the vaccinated cohort, consistently had lower rates of all negative control outcomes assumed to be markers of health-seeking behaviour. We hypothesize that people with psychosocial surplus, high conscientiousness, and confidence in the healthcare system are prone to follow vaccination recommendations. As shown in Table 1, the vaccinated population had more SARS-CoV-2 tests before study inclusion, in accordance with health-seeking behaviour leading to a greater test frequency and consequently a higher rate of positive SARS-CoV-2 tests in the vaccinated than the non-vaccinated population. The congruence in IRRs for negative control outcomes indicative of health-seeking behaviour and positive SARS-CoV-2 tests indicated that the same confounders likely affected these effect estimates.

Our results also support our hypothesis that it is difficult to remove confounding in estimate of the effect of vaccination on mortality. Mortality after SARS-CoV-2 infection and several non-infectious diseases (negative control cohorts) was substantially lower in vaccinated than non-vaccinated individuals. SARS-CoV-2 vaccination might have decreased the risk of COVID-19 in the negative control cohorts and thereby decreased mortality. However, while the vaccine likely protects against death, a genuine vaccine effect is unlikely to explain the extensively diminished mortality observed in the vaccinated individuals in the negative control cohort, given that only a small fraction of those individuals was infected with SARS-CoV-2 near the time of diagnosis of the non-infectious diseases that led to their inclusion in the negative control cohorts. Furthermore, the estimates changed only marginally when time was censored at the date of SARS-CoV-2 infection and the date of vaccination in the non-vaccinated population. Because of frequent testing for SARS-CoV-2 in the study period, particularly for inpatient hospital admissions, a large fraction of the individuals who died in the study period were unlikely to have died with unrecognized SARS-CoV-2 infection. We therefore presume that the lower mortality in the vaccinated compared to the non-vaccinated population stems from confounding. We believe that healthy vaccine bias might explain much of the vaccine effectiveness observed in observational studies.

We did not observe substantial differences in comorbidity between the vaccinated and non-vaccinated cohorts. However, we assume that the non-vaccinated cohort might have had comorbidities not registered in the national Danish health registers. Furthermore, the non-vaccinated cohort might have had poor health associated with alcohol use and smoking which increase mortality in people with severe non-infectious diseases.^32–34

Several observational studies have examined SARS-CoV-2 vaccine effectiveness, but whether these studies were able to completely mitigate confounding is unclear; moreover, the studies did not include analyses of negative control outcomes.^12,35–39 Confounding in observational studies is not unique to SARS-CoV-2 vaccination and has been described in observational studies of influenza vaccines.^9,11,13,14 The confounding described in our study might potentially explain the unspecific effects of vaccination described in previous observational studies.⁴⁰ Confounding of vaccine effectiveness due to differences in health-seeking behaviour may be less pronounced in studies using the test negative case–control design, where the study population is restricted to people who have contacted healthcare to get tested.

Our study has several strengths including a large national population-based cohort with no losses to follow-up; complete data on hospitalization and vaccine status; and extensive data on SARS-CoV-2 testing.

Nevertheless, several study limitations must be considered. In Denmark, the testing strategy changed substantially in March 2022, when recommendations advised testing for SARS-CoV-2 only in special vulnerable populations, and home-based SARS-CoV-2 testing increased, thus potentially influencing our estimates of rates of positive tests for SARS-CoV-2. In addition, adjustment for socioeconomic factors and other lifestyle factors might have decreased confounding. We did not have access to such data, and the data are typically unavailable in observational studies of vaccine effectiveness based on healthcare registries. In addition, we cannot exclude the possibility of some misclassification of diagnoses in Danish National Patient Registry, but we assume that such misclassification would be non-differential, thus resulting in low potential effects on relative risk estimates. In addition, the included confounders might have been under-registered, thereby leading to residual confounding. In addition, we did not attempt to prevent bias in the analysis of the effect of the vaccine on mortality for any collider bias due to conditioning on infection. We used this design because our aim was to demonstrate bias in methods currently used for vaccine effectiveness studies based on observational data, which also use this approach. Nonetheless, if we were attempting to estimate vaccine effectiveness, we would not want to condition on infection but instead would aim to examine the overall effect of vaccination on death from COVID-19, which would require infection and could also include any effects of the vaccine in preventing infections.

Conclusions

We conclude that negative control methods indicate the presence of substantial confounding in observational studies of SARS-CoV-2 vaccine effectiveness, and such studies should therefore be interpreted critically. Our findings indicate a need for randomized vaccine efficacy studies after the emergence of new SARS-CoV-2 variants and the rollout of multiple booster vaccines.

Patient and Public Involvement Statement

It was not appropriate or possible to involve patients or the public in the design, or conduct, or reporting, or dissemination plans of our research.

Ethical Approval

This analysis was performed as part of national surveillance efforts under the authority of the Danish National Infectious Disease Control Institute at the Statens Serum Institut. The analysis was approved by the Danish Data Protection Agency (permission no. 21/04383). According to Danish regulations, national surveillance activities relying solely on register information do not require individual consent or approval from an ethics committee.

Author Contributions

All authors made a significant contribution to the work reported, whether that is in the conception, study design, execution, acquisition of data, analysis and interpretation, or in all these areas; took part in drafting, revising or critically reviewing the article; gave final approval of the version to be published; have agreed on the journal to which the article has been submitted; and agree to be accountable for all aspects of the work.

Disclosure

MMT has received travel grants outside this work from GlaxoSmithKline Pharma A/S and has received funding from Copenhagen University Hospital – Rigshospitalet. Professor Henrik Sørensen reports that The Department of Clinical Epidemiology, Aarhus University, receives funding for other studies in the form of institutional research grants to (and administered by) Aarhus University. None of these studies has any relation to the present study. The authors report no other conflicts of interest in this work.

Data Sharing Statement

The ethical approval of this study from the Danish Data Protection Agency states that the data that have been used in this article cannot be shared publicly.

Additional information

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.