Pandemic influenza vaccine & narcolepsy: simulations on the potential impact of bias

ABSTRACT

Several studies have identified an association between Pandemrix^TM, an AS03 adjuvanted pandemic influenza A(H1N1) vaccine, and narcolepsy, a rare and under-diagnosed sleep disorder with a median onset-to-diagnosis interval of ten years. This paper reviews potential sources of bias in published studies and aims to provide, through simulation, methodological recommendations for assessment of vaccine safety signals. Our simulation study showed that in the absence of an association between the vaccine and the outcome, presence of detection bias and differential exposure misclassification could account for elevated risk estimates. These may play a major role, particularly in alert situations when observation times are limited and the disease has a long latency period. Estimates from the case-control design were less inflated than those from the cohort design when these biases were present. Overall, these simulations provide useful insights for the design and interpretation of future studies.

KEYWORDS:

• Narcolepsy
• Pandemrix
• Bias
• Simulation
• Pandemic Influenza
• Vaccine Safety

Background

In August 2010, case reports linking the occurrence of narcolepsy in children aged 5–19 years to an AS03 adjuvanted H1N1pdm09 (pH1N1) vaccine, Pandemrix^TM (GlaxoSmithKline, Middlesex, UK) were published in Finland and Sweden [1,2]. In the European Union, Pandemrix was widely used, with over 30 million doses administered. Coverage was particularly high in the Nordic countries [3]. Following reports from Sweden and Finland, the European Medicines Agency initiated a review procedure [4] which eventually led to the restriction of indication for Pandemrix [5].

Narcolepsy is a chronic sleep disorder that is severely debilitating. The dysregulation of the sleep–wake cycle is caused by the destruction of hypocretin forming neurons in the hypothalamus, which is thought to result from an auto-immune process [6]. Symptoms include excessive daytime sleepiness (EDS) and cataplexy [7]. Symptoms usually emerge gradually and can initially be non-specific. Consequently, symptoms can be attributed to other diagnoses resulting in a delay of narcolepsy diagnosis and treatment [8–11]. Despite significant improvements in the speed and accuracy of narcolepsy diagnosis [10–12], a recent study found that the median delay between onset and diagnosis remains approximately 10 years [9].

As of May 2015, eight epidemiological studies testing the association between Pandemrix and clusters of narcolepsy cases [13–21] have been published and reporting risk estimates ranging from 1.6 to 14.4. An overview of the main characteristics of these studies is presented in Table 1. Generally, published studies were meticulous in their methods and applied sensitivity analyses to evaluate the presence of biases. Nonetheless, studies were inevitably observational and, as studies were mostly initiated rapidly after the signal emerged, they had limited time for case capture. Combined with the often nonspecific symptoms and onset of narcolepsy resulting in delayed diagnosis these studies are particularly prone to bias. Five years after the original signal emerged it remains unclear if and how potential sources of bias affected the estimates from the association studies. Consequently it is still unknown what the exact association between Pandemrix and narcolepsy is [22,23].

Table 1. Main characteristics testing association between narcolepsy and Pandemrix.

Ref	Year	Country, region	Age range	Case definition	Index date	Exclusion criteria	Primary study period	Dealt with confounding	Dealt with detection bias	Relative rate reported
Retrospective cohort studies
[18]	2012	Finland	4–19 years	BC definition level 1–3	First contact for EDS	-	01/01/2009 to 15/08/2010	No	Limit observation period to start media attention. Sensitivity analysis on start professional attention	RR:12 · 7 (95% CI: 6 · 1–30 · 8)
[21]	2012/2014	Ireland		BC definition level 1–3	First contact for EDS	Those with symptom onset before April 2009	01/04/ 2009 to 31/12/2010	No	Sensitivity analysis considering period before the increased media attention in Sweden and Finland	RR: 13 · 0 (95% CI: 4 · 8–34 · 7)
[14]	2011	Sweden; Stockholm, Skane, Vastra Gotaland, Ostergotland		Presence diagnosis code (non-validated)	Diagnosis	Those with onset before October 2009	01/10/2009 to 31/12/2010	No	No	HR: 4 · 19 (95% CI: 1 · 76–12 · 1)
[17]	2013	Sweden; Stockholm, Skane, Vastra Gotaland, Ostergotland)		Presence diagnosis code (non-validated)	Diagnosis	Prevalent cases (looking back 5 years)	01/10/2009 to 31/12/2010	Adjusted for age (5-year bands), gender, county, education, income, secondary health care use, pregnancy status & presence of diagnoses (ICD-10), ethnicity	No	HR: 2 · 92 (95% CI: 1 · 78–4 · 79)
Case–control studies
[20]	2013	France		BC definition level 1–3	Diagnosis	Non-consenting cases, cases with onset before 1 January 2005, Vaccinated cases with onset before vaccination	01/10/2009 to 30/04/2011	Controls were matched on age, gender and geographic location	Sensitivity analyses were performed that considered as index date (i) the date of referral for polysomno-graphy – MSLT; and (ii) the date of first symptom onset	OR: 4 · 55 (95% CI: 2 · 34–8 · 88)
[15]	2012	Finland, Sweden, Norway, Italy, Denmark, UK, Netherlands, France	<19 years	BC definition level 1–4	MSLT referral	-	01/04/ 2009 to 30/06/2010	Controls were matched on year of birth, sex and index date (i.e. the date of onset of narcolepsy) and in Norway, Italy and UK also by region/practice	Sensitivity analyses with different time periods (before media & professional attention); analysis of changes in lag times	Signal-generating countries OR (children and adolescents): 14 · 2 (95% CI: 2 · 5–infinity)
										Non-signal-generating countries OR (children and adolescents): 1 · 6 (95% CI: 0 · 5–6 · 1)
[19]	2013	England	4–18 years	International classification of sleep disorders criteria (definite & probable)	EDS onset	Cases with onset before 2008 and/or diagnosis after July 2011	01/10/2009 to 31/12/2010	Adjusted for age, time period & clinical conditions that were indications for pandemic vaccination	Choice of index date, sensitivity analyses included diagnosed case only and varying of index date (first healthcare contact or index date)	OR: 14 · 4 (95% CI:4 · 3–48 · 5)
[13]	2011	Sweden	0 to<19 years	American Academy of Sleep Medicine criteria for narcolepsy with cataplexy	EDS onset	Cases without cataplexy	01/01/2009 to 31/12/2010	No	Choice of index date, analysis by different time windows after vaccination (3 months).	RR: 6 · 6 (95% CI: 23 · 1–14 · 5)
[16]	2014	Quebec province, Canada	>6 months (on 01/07/2009)	BC definition level 1–3	Date of onset, determined throughquestionnaire and interview	Those with symptom onset before January 2009	01/01/2009 to 31/12/2010	Adjusted for age, gender, seasonality and circulation of H1N1 virus (cohort). Matched by age and gender (case–control)	Sensitivity analyses including different observation periods to account for circulation virus and vaccination	RR (cohort): 4 · 58 (95% CI: 1 · 59–11 · 77)
										OR (CC): 1 · 48 (95% CI: 0 · 37–7 · 03)
										RR (SCCS): 2 · 07 (95% CI: 0 · 70–6 · 17)

It is not unthinkable that a similar scenario could unfold in the future, that is, a safety signal involving a difficult to diagnose condition with a delayed onset is linked to exposure with a new vaccine. Indeed, a similar situation has occurred in the past, when clusters of cases of Guillain–Barré syndrome were detected after the introduction of a new swine flu vaccine [24]. Using the example of narcolepsy and Pandemrix, we explore, in the absence of a formal hypothesis, the potential impact of two sources of bias that are likely to occur in similar scenarios.

Detection bias. The first source of bias is a type of selection bias. Awareness of a potential association between narcolepsy and vaccination amongst physicians and the general public could result in earlier diagnosis for vaccinated cases compared with unvaccinated cases, making vaccinated cases more likely to be included in observational studies with limited observation time [15]. We refer to this as ‘detection bias’.

Differential exposure misclassification. A second source of bias we consider is a form of recall bias, in which the onset of symptoms is misattributed, resulting in misclassification of onset dates to the period following vaccination. As narcolepsy symptoms often develop gradually and onset of symptoms is not always clearly identifiable, studies into narcolepsy are particularly prone to recall bias. We hypothesize that recalling onset of EDS with knowledge of a putative association between vaccination and narcolepsy could lead a patient to recall that symptoms started after vaccination [25]. We refer to this as ‘differential exposure misclassification’.

Methods

We considered the impact of detection bias and differential exposure misclassification as defined above on the association measure between Pandemrix and narcolepsy.

Simulation

We simulated a population of 100,000 subjects <19 years of age on 1 April 2009 to mimic the signal-generating population. We subsequently simulated dates of birth and death (based upon average lifespans in Western Europe) to create a simulated lifetime for each subject. EDS onset dates were assigned over the lifespan of subjects based upon the reported age and gender specific incidence rates of narcolepsy with cataplexy onset [26]. Given these EDS onset dates, initial narcolepsy diagnosis dates were assigned using a random value drawn from a distribution of narcolepsy onset-to-diagnosis intervals which was assumed to have a gamma distribution chosen to mimic the distribution of onset-to-diagnosis intervals reported in the literature: a median of 10 years with a range of 0–40 years [11]. Additionally, as the underlying onset-to-diagnosis interval in children is potentially shorter [10], alternate gamma distributions with medians of 3 (range 0–13) and 7 (range 0–27) years were also used. All onset-to-diagnosis intervals were simulated to be at least 40 days long.

Overall vaccination coverage in this population was simulated at 25, 50 and 75%. Vaccination dates were assigned independent of the age of a subject using a beta distribution of administration times mimicking real-life Pandemrix administration dates between 12 October 2009 and 12 February 2010 [27].

A null association (RR = 1) was assumed for the actual relation between vaccine exposure and outcome.

Detection bias. Reduction in the onset-to-diagnosis interval was applied only to vaccinated cases for whom initial diagnosis would occur after the date of media attention (simulated to be 15 August 2010). If EDS onset occurred before 15 August 2010, the reduction was applied only to the interval from 15 August 2010 to the initial date of diagnosis. The date of narcolepsy diagnosis was reset in this way using values drawn from logit-normal distributions with medians of 30, 60, and 90% (see Figure A1 in appendix) to produce reductions of the interval, with the restriction that the interval still was at least 40 days. Data with no reduction (i.e. 0% reduction) in the interval were also simulated (Figure 1).

Figure 1. Application of detection bias.

Differential exposure misclassification. Misattribution of EDS onset dates to the period following vaccination was applied with probability equal to values drawn from logit-normal distributions with medians of 30 and 60% (Figure 1) to subjects who were diagnosed with narcolepsy after vaccination and after the start of media attention. In this case the onset date was reset to a random date between the vaccination date and the minimum of diagnosis date and vaccination date plus 180 days, based upon the 6 month risk period used by Miller et al. in their self-controlled case series analysis [19]. Data with no misattribution of onset dates were also simulated (Figure 2).

Figure 2. Application of differential exposure misclassification.

We simulated nine combinations of the underlying population settings: gamma scale (baseline onset-to-diagnosis interval, three different values) and vaccination coverage (three values), to which we applied 12 combinations of the simulated sources of bias: detection bias (four values), and differential exposure misclassification parameters (three values) for a total of 108 combinations of simulation parameters. Variation in the underlying population settings (baseline onset-to-diagnosis interval and vaccination coverage) was conducted in the absence of a hypothesis regarding the impact of these changes on effect estimates.

Analysis

The association between vaccination and narcolepsy in children aged 4 to <19 years during the study period was analyzed using dynamic cohort and case–control designs. In the primary analyses a case capture (study period) of 1 April 2009 to 1 December 2010 was used in line with several published studies. We calculated absolute incidence rates in 6-month periods and calculated case counts during exposed and unexposed person time to investigate how incidence would change over time in the presence of detection bias. We additionally calculated the number of onset dates in exposed and unexposed person time at each level of each of the two bias parameters. In the comparative cohort analysis, the incidence rate of narcolepsy was compared between dynamic cohorts of vaccinated and non-vaccinated persons. All person time after the date of vaccination was considered exposed, whereas the entire case-capture period of non-vaccinated persons as well as the pre-vaccination time in vaccinated subjects contributed to non-exposed person time. Rate ratios were calculated based on Poisson regression. In the case–control analysis, cases were matched to 10 controls on sex, age in years and onset date. Odds ratios were calculated using conditional logistic regression.

We conducted several analyses to investigate the effects of different design choices and ways to mitigate bias. All sensitivity analyses were conducted using vaccination coverage of 50% and the baseline onset-to-diagnosis interval distribution described in literature with median 10 years, range 0–40 years. To study the effect of the length of case-capture period, analyses with observation periods as long as 50 years were conducted. To study the effect of exclusion of cases possibly affected by awareness of a putative association, in one of the settings we excluded the cases with onset dates and diagnosis dates after 15 August 2010. Each of these sensitivity analyses was conducted in the absence of a hypothesis.

For each set of simulation parameters, 500 replications were analyzed, each producing an estimate and 95% confidence interval. Reported results are the exponentiated median of these 500 estimates calculated on the log scale and medians of the lower and upper confidence limits. All analyses were conducted using SAS 9 · 2.

Results

Application of onset-to-diagnosis interval reduction (detection bias) and differential exposure misclassification over three coverage rates and three baseline onset-to-diagnosis intervals increased the number of narcolepsy onset dates observed in the study period. Figure 3 shows, for exposed and unexposed children, the number of onset dates associated with narcolepsy diagnosed cases in scenarios with different percentages of differential exposure misclassification (columns), vaccination coverage (rows) and levels of detection bias (x-axis in each plot), using a baseline onset-to-diagnosis interval with a median of 10 (range 0–40) years. The number of observed narcolepsy onset dates increases at approximately the same rate in exposed and unexposed person time with an increasing detection bias in the absence of differential exposure misclassification (within column 1, Figure 3) except when vaccine coverage is 25% in which case no onset dates are observed in exposed person time. With the introduction of differential exposure misclassification in exposed subjects, new narcolepsy diagnoses occur more often in post-vaccination person time. The number of onset dates within unexposed person time also increases with increased reduction in the onset-to-diagnosis interval because, in these cases, the bias is being applied to vaccinated cases who experienced onset prior to vaccination. Figure 4 shows the effects of reduction of EDS onset-to-diagnosis date on the shape of incidence rates over calendar time in this cohort of 0–19 year olds in 2009. With a reduction of 60 or 90% in lag time, a clear peak in incidence of narcolepsy diagnoses occurs after media attention. These rates then return to the baseline rate or fall below the baseline rate because of depletion of cases through early diagnosis. The primary study period of 1 April 2009 to 1 December 2010 is indeed a period of marked increase in newly diagnosed cases with reduction in time from onset-to-diagnosis.

Figure 3. Case Counts (Onset dates) occurring in exposed and unexposed person time.

Figure 4. Incidence of new narcolepsy diagnoses over time.

Table 2 shows the results of cohort and case–control analyses of all 108 different parameter settings.

Table 2. Relative risks and odds ratios in primary cohort and case–control analyses.

Baseline onset-to-diagnosis interval	Differential exposure misclassification bias	Detection bias	Coverage = 25%		Coverage = 50%		Coverage = 75%
			Cohort RR (95% CI)	Case–control OR (95% CI)	Cohort RR (95% CI)	Case–control OR (95% CI)	Cohort RR (95% CI)	Case–Control OR (95% CI)
Median3 years, range 0–13 years (Gamma ScaleParameter = 2)	0	0	0.42 (0.16, 1.12)	0.97 (0.34, 2.77)	0.38 (0.19, 0.77)	0.99 (0.47, 2.11)	0.31 (0.17, 0.57)	1.02 (0.53, 1.94)
		30	0.57 (0.25, 1.32)	1.31 (0.54, 3.13)	0.5 (0.27, 0.93)	1.19 (0.61, 2.31)	0.42 (0.25, 0.71)	1.14 (0.65, 1.98)
		60	1.05 (0.57, 1.92)	2.06 (1.06, 4.04)	0.82 (0.52, 1.31)	1.69 (1.03, 2.76)	0.61 (0.41, 0.91)	1.33 (0.87, 2.04)
		90	3.61 (2.62, 5.02)	5.03 (3.5, 7.17)	2.15 (1.68, 2.76)	2.83 (2.18, 3.69)	1.32 (1.07, 1.63)	1.43 (1.19, 1.68)
	30	0	1.09 (0.57, 2.08)	2.1 (1.02, 4.21)	0.99 (0.6, 1.62)	1.74 (1.02, 2.93)	0.88 (0.56, 1.4)	1.43 (0.88, 2.31)
		30	1.51 (0.88, 2.6)	2.6 (1.37, 4.87)	1.4 (0.89, 2.18)	2.08 (1.29, 3.31)	1.11 (0.74, 1.66)	1.58 (1.04, 2.42)
		60	2.75 (1.76, 4.27)	3.92 (2.42, 6.34)	2.27 (1.59, 3.27)	2.73 (1.85, 4.01)	1.8 (1.29, 2.52)	1.89 (1.34, 2.67)
		90	8.09 (6.11, 10.73)	8.11 (5.97, 11.05)	5.27 (4.18, 6.63)	4.57 (3.58, 5.83)	3.39 (2.76, 4.17)	2.55 (2.06, 3.17)
	60	0	1.81 (1.07, 3.09)	2.93 (1.63, 5.15)	1.64 (1.06, 2.55)	2.2 (1.37, 3.51)	1.54 (1.02, 2.31)	1.71 (1.12, 2.64)
		30	2.61 (1.63, 4.23)	3.62 (2.15, 6.08)	2.34 (1.58, 3.48)	2.69 (1.79, 4.07)	2.07 (1.42, 3.02)	1.95 (1.32, 2.91)
		60	4.71 (3.2, 6.9)	5.41 (3.55, 8.28)	4.1 (2.94, 5.75)	3.76 (2.64, 5.35)	3.43 (2.48, 4.79)	2.54 (1.8, 3.57)
		90	14.36 (10.88, 18.95)	12.49 (9.24, 16.84)	10.39 (8.13, 13.29)	7.48 (5.78, 9.69)	7.18 (5.71, 9.04)	4.19 (3.30, 5.35)
Median 7 years, range 0–27 years (Gamma ScaleParameter = 4)	0	0	0.35 (0.05, 2.92)	0.9 (0.11, 9.73)	0.31 (0.07, 1.38)	0.97 (0.22, 4.5)	0.26 (0.08, 0.83)	1.03 (0.31, 3.47)
		30	0.53 (0.12, 3.11)	1.16 (0.19, 7.65)	0.43 (0.13, 1.41)	1.29 (0.36, 4.33)	0.37 (0.14, 0.98)	1.16 (0.41, 3.32)
		60	0.93 (0.31, 2.95)	2.18 (0.63, 7.85)	0.75 (0.31, 1.75)	1.74 (0.72, 4.22)	0.58 (0.28, 1.19)	1.36 (0.63, 2.93)
		90	4.13 (2.58, 6.67)	5.71 (3.37, 9.75)	2.24 (1.57, 3.22)	2.99 (2.03, 4.38)	1.27 (0.93, 1.72)	1.74 (1.26, 2.4)
	30	0	1.88 (0.75, 4.78)	3.1 (1.1, 9.02)	1.73 (0.8, 3.74)	2.35 (1.02, 5.38)	1.5 (0.72, 3.11)	1.76 (0.83, 3.75)
		30	2.71 (1.19, 6.25)	3.89 (1.56, 9.89)	2.16 (1.11, 4.27)	2.75 (1.31, 5.71)	1.89 (0.98, 3.59)	2.01 (1.02, 3.94)
		60	4.73 (2.47, 9.05)	5.57 (2.75, 11.61)	3.79 (2.16, 6.73)	3.74 (2.06, 6.81)	3.06 (1.78, 5.26)	2.39 (1.38, 4.14)
		90	14.96 (9.84, 22.58)	12.7 (8.19, 19.61)	8.43 (6.05, 11.7)	6.44 (4.57, 9.12)	5.12 (3.83, 6.85)	3.28 (2.44, 4.44)
	60	0	3.45 (1.57, 7.59)	4.42 (1.82, 10.78)	3.29 (1.66, 6.5)	3.24 (1.59, 6.66)	3.01 (1.54, 5.83)	2.37 (1.19, 4.75)
		30	5.12 (2.56, 10.3)	5.77 (2.62, 12.43)	4.42 (2.41, 8.24)	3.83 (2.06, 7.45)	3.94 (2.14, 7.31)	2.89 (1.51, 5.52)
		60	8.97 (4.97, 16.34)	8.55 (4.48, 16.2)	8.44 (4.84, 14.56)	5.79 (3.27, 10.28)	6.63 (3.85, 11.22)	3.74 (2.14, 6.64)
		90	32.43 (20.97, 49.31)	22.68 (14.57, 35.31)	20.09 (13.87, 29.04)	12.95 (8.84, 19.04)	12.69 (9.1, 17.65)	6.52 (4.44, 8.81)
Median 10 years, range 0–40 years (Gamma ScaleParameter = 6)	0	0	0 (0, NA)	0 (0, NA)	0.31 (0.04, 4.43)	1.03 (0.16, 12.32)	0.22 (0.03, 1.83)	1.15 (0.18, 6.7)
		30	0.48 (0.06, 9.09)	1.12 (0.11, 61.51)	0.36 (0.05, 2.98)	1.15 (0.2, 8.23)	0.34 (0.08, 1.55)	1.27 (0.26, 5.98)
		60	1.03 (0.2, 6.25)	2.1 (0.34, 15.68)	0.72 (0.21, 2.65)	1.83 (0.5, 6.71)	0.55 (0.19, 1.57)	1.43 (0.45, 4.46)
		90	4.37 (2.35, 8.25)	6.08 (2.93, 12.35)	2.24 (1.39, 3.62)	2.99 (1.79, 5.0)	1.26 (0.84, 1.88)	1.73 (1.14, 2.62)
	30	0	2.34 (0.67, 8.67)	3.76 (0.9, 16.51)	2.29 (0.8, 6.82)	2.91 (0.93, 8.74)	2.02 (0.76, 5.43)	1.95 (0.7, 5.61)
		30	3.82 (1.22, 11.13)	4.96 (1.49, 16.85)	2.87 (1.13, 7.32)	3.41 (1.25, 9.25)	2.77 (1.11, 6.53)	2.19 (0.87, 5.58)
		60	6.19 (2.48, 15.72)	7.47 (2.67, 20.57)	5.65 (2.56, 12.36)	4.52 (2.0, 10.46)	4.08 (1.92, 8.38)	2.76 (1.26, 5.85)
		90	19.32 (11.4, 33.15)	15.55 (8.85, 27.5)	10.36 (6.83, 15.73)	7.55 (4.86, 11.72)	6.11 (4.24, 8.82)	3.76 (2.56, 5.47)
	60	0	4.76 (1.66, 13.52)	5.78 (1.81, 19.46)	4.31 (1.68, 10.74)	4.16 (1.54, 11.17)	4.54 (1.69, 11.83)	2.85 (1.05, 7.6)
		30	6.82 (2.71, 17.06)	7.24 (2.54, 21.47)	6.44 (2.67, 15.22)	5.22 (2.04, 12.88)	5.71 (2.34, 13.4)	3.17 (1.32, 7.97)
		60	12.92 (5.7, 29.72)	11.3 (4.53, 27.27)	10.63 (5.08, 22.61)	7.73 (3.44, 16.83)	8.65 (4.14, 18.89)	4.83 (2.18, 10.74)
		90	46.02 (25.69, 82.4)	32.09 (17.78, 58.46)	28.4 (17.13, 47.12)	16.98 (10.15, 28.1)	17.29 (11.15, 26.99)	7.74 (4.98, 12.12)

Because a 10-year onset-to-diagnosis interval has been reported in the literature, we have chosen to illustrate our results using underlying populations with this onset-to-diagnosis interval and the intermediate vaccine coverage of 50%.

Using a cohort analysis on this underlying population to which the maximum reduction of time from EDS onset-to-diagnosis (90%) has been applied in the absence of differential exposure misclassification produced a median RR of 2.24 (95% CI: 1.39, 3.62). In case–control analysis, the same simulation parameter settings produced an OR of 2.99 (95% CI: 1.79, 5.00) (Table 2).

In the absence of a reduction in the EDS onset-to-diagnosis interval, differential exposure misclassification resulted in a RR of 4.31 (95% CI: 1.68, 10.74) when vaccination coverage was 50% and the EDS date was attributed to the post-vaccination period with a median probability of 60% for vaccinated cases. In the case–control analysis, the same simulation parameter settings produced an OR of 4.16 (95% CI: 1.54, 11.17) (Table 1).

When combining the effect of detection bias and differential exposure misclassification, the estimates were higher in cohort analyses than in case–control analyses as the biases became more pronounced. In the most extreme scenario, with a median 90% reduction in the onset-to-diagnosis interval in vaccinated cases and a median probability of differential misclassification equal to 60% in vaccinated cases, we found a RR of 28.4 in the cohort analysis (95% CI: 17.13, 47.12). The same parameter settings produced an OR of 16.98 (95% CI: 10.15, 43.85) in the case–control analysis (Table 2). In the absence of either source of bias, median RR estimates from the cohort analysis for all scenarios were <1 when observation time was limited. However, with extension of observation time up to 25 years, the RR was estimated to equal to the simulated RR of 1.

Results from case–control analyses were less inflated than those from cohort analyses when detection bias and differential exposure misclassification were present. For both case–control and cohort designs, increased vaccination coverage and a shorter baseline onset-to-diagnosis interval lead to RR estimates closer to the true rate of one when biases are present (Table 1).

Extension of the case-capture period reduces the bias (Figure 5). With each extension, the rate of narcolepsy in vaccinated subjects converges toward the background rate. As illustrated in Figure 4, reduction in time from onset-to-diagnoses leads to incidences greater than the background rate in the period following awareness of the association in vaccinated cases, followed by reduction in the incidence rate to levels below the background rate.

Figure 5. Relative rates of narcolepsy with extended observation periods.

When cases with an onset date after 15 August 2010 were excluded, the RR was 1.87 (95% CI: 1.15, 3.05) in cohort analyses for the extreme setting of detection bias in the absence of differential exposure misclassification. Similarly, the RR was 4.45 (1.76, 11.67) with exclusion of cases with onset after 15 August 2010 at the most extreme setting of differential exposure misclassification in the absence of detection bias. Exclusion of cases with EDS onset dates after media attention, with a 90% reduction in the onset-to-diagnosis interval and a 60% probability of differential exposure misclassification, produced an RR of 27.10 (95% CI: 16.52, 44.11) while the estimate was 28.4 (95% CI: 17.13, 47.12) when these cases were not excluded. Exclusion of all cases with a diagnosis of narcolepsy after media attention resulted in estimates <1 and confidence intervals including 1 for all parameter settings. Excluding these cases nullified the effect of differential misclassification bias because only those cases diagnosed after media attention were simulated to misattribute their date of EDS onset to the period following vaccination.

Discussion

Our results indicate that, in the absence of a real association between Pandemrix and narcolepsy, the presence of detection bias or differential exposure misclassification elevates risk estimates.

In the absence of either source of bias, median RR estimates from the cohort analysis for all scenarios were less than the expected value of 1. Our explanation for this observation is as follows. The study observation period is limited and the interval between onset and diagnosis can be longer than the study observation time, therefore, as diagnosis is the criteria for case inclusion, a number of cases with onset within the observation period will not be included as cases. However, exposed and unexposed person time within the cohort is fixed. When we analyzed all cases with onset within the observation period regardless of their diagnosis date, the RR = 1. In the absence of either bias, using diagnosis dates for case capture, an observation period as long as 25 years would be necessary to obtain the true RR of 1.

We found that biased attribution of EDS onset (differential exposure misclassification) has a greater impact on the estimates than a reduction in the EDS onset-to-diagnosis interval (detection bias) both in the cohort and case–control designs. While detection bias increases the relative risk estimates, the effect is not discernible until the onset-to-diagnosis interval is so reduced that many additional cases can be detected in a short observation period. The simultaneous presence of detection bias and differential exposure misclassification increases RRs more rapidly than could be expected by the effect size of each bias in isolation.

In an attempt to exclude detection bias, several published studies limited their primary observation period for EDS onset to the period before media attention [15,18] or included sensitivity analyses using such a reduced study period [21]. Additionally, studies used primary index dates that were thought to be less susceptible to such a bias including onset of symptoms [14,19], first contact with health care [18,21] or referral to specialist care [15,20]. In line with observations from our simulations, limiting analysis to subjects with an onset date prior to media attention will not eliminate the effect of detection bias, as all patients need to be diagnosed to be included, which is where the bias arises. To illustrate this, when limiting cases to those with an EDS date before media attention, Nohynek et al. found that the RR increased from 11.4 to 12.7 [18] and O’Flanagan et al. found that the RR increased from 13.0 to 14.5 [21]. As only diagnosed subjects can be included as cases, detection bias will be unavoidable if the onset-to-diagnosis interval is shorter in vaccinated individuals. The only way to circumvent the combined effects of detection bias and differential exposure misclassification would be to select only patients diagnosed before media attention. This will result in limited observation time and limited case inclusion as illustrated by our simulations and as was shown in the VAESCO study (13). We are not aware of any existing statistical methods to control for detection bias although quantitative bias analysis could adjust for hypothesized biases [28].

With limited observation time, we found that, in the presence of detection bias and differential exposure misclassification, estimates from the case–control design are less inflated than those from the cohort design. The resilience of the case–control in this scenario has several causes: the outcome is rare and the pool of controls, matched only by sex and age at onset, is large; also, the invariability of exposed person time, which is limited by observation time and vaccine coverage in the cohort approach, is avoided. Additionally, in this simulated scenario, we were able to sample controls from the same population as the cases and to assess their exposure without error, thereby avoiding the most problematic sources of bias in case–control studies. The only study to date in which data were analyzed using both a case–control and a cohort design found lower estimates in the case–control than in the cohort design [16]. Applying these findings to the interpretation of all published studies, however, presents a challenge as each study differed not only in design choice but in many other ways including underlying population, diagnostic practices, inclusion and exclusion criteria, and many others. In general, however, estimates from case–control studies were similar to those from those cohort studies in which diagnosis was used as an index date. Those cohort studies which used onset of symptoms as an index date produced much higher estimates, suggesting presence of bias, particularly differential exposure misclassification, a true association in those populations, or both. However, given the complexity of the interplay of design choices and underlying populations, a meaningful comparison between designs implemented in published studies is not feasible.

Increased vaccination coverage reduced the bias in cohort and case–control analyses. In cohort analyses, this is explained by an increase in the person time denominator for vaccinated cases with a smaller increase in events and, simultaneously, a decrease in the person time denominator for unvaccinated cases with a smaller decrease in the number of events. In case–control analyses, this could be attributed to a greater probability of matching to vaccinated controls as vaccination coverage increases, leading in turn to more informative strata in a conditional analysis.

When a shorter interval from onset-to-diagnosis was assumed, the impact of simulated biases was less pronounced. This is because of the fact that, with a shorter onset-to-diagnosis interval, more cases, whether vaccinated or not, are being captured during the study period.

We chose to simulate only those sources of bias for which data in the absence of a vaccine safety signal exists and for which simulated variables could be modified to mimic the bias. Our simulations therefore do not reflect all of the biases that could potentially affect estimates of an association between Pandemrix and narcolepsy. For example, it is possible that non-vaccinated cases also experienced a reduction in the onset-to-diagnosis interval because of increased awareness of narcolepsy. However, inclusion of additional simulation parameters such as this would have required the making of additional assumptions for which we had no basis in published data. By focusing on biases that could be evaluated without making untenable assumptions, these simulations provide insights that can improve rapid evaluation of vaccine safety signals by decision makers. There were several uncertainties, including the true background rate of narcolepsy and the true interval between onset of symptoms and diagnosis, for which we made assumptions in order to conduct our simulations. The validity of these assumptions will ultimately determine the robustness of our simulations.

The introduction of a new vaccine, or an existing vaccine in new populations, requires the assessment of vaccine safety. Large numbers of people can be exposed in a relatively short period providing a challenge to real-time safety surveillance. In such situations, as illustrated by the experience with Pandemrix and narcolepsy during the 2009/2010 H1N1 pandemic, it can be difficult to determine if a safety finding is a true association or not. Despite these challenges, the timely and accurate assessment of potential associations between adverse events and vaccination are crucial to ensure vaccine safety and maintain the public’s confidence. We believe that our simulations provide useful insights for the design and interpretation of future studies. Importantly, our results illustrate that in future analyses of safety signals for diseases with long latency periods for which observation times are limited the effect of limited case capture together with fixed person time denominators should be recognized. Similarly, the changes in exposed and unexposed person time denominators with changing vaccination coverage should also be taken into account. As we have shown, the case–control design provides less biased estimates in these circumstances as it does not require the calculation of person time. Moreover, our simulations illustrate the importance of not only understanding background rates of adverse events of special interest prior to vaccination campaigns, but also having insight in the background onset-to-diagnosis interval.

Recommendations:

• Because rapid assessment of a vaccine safety signal, by definition, means limited case-capture time, not only the background incidence of events of interest but the background onset-to-diagnosis interval should be understood for proper interpretation of risk estimates.

• The impact of differential exposure misclassification in these simulations underlines the need for accurate and linkable vaccine registries as well as blinded assessment of cases.

• When person-time is fixed and the outcome is rare, a case–control design is more resilient to bias and should be considered.

Population cohorts should continue to be followed over time to monitor how rates of narcolepsy change following the H1N1 pandemic. If these biases were indeed present, we would expect to see incidence eventually fall back to or even below the background rate.

To conclude, our results indicate that, in the absence of a real association between the vaccine and narcolepsy, presence of detection bias and differential exposure misclassification could account for elevated RRs in vaccinees in association studies. While this does not exclude a real increased risk of narcolepsy following Pandemrix, it is possible that the levels of increased risk observed were at least partially due to bias.

Expert commentary

When the narcolepsy signal emerged in Sweden and Finland in 2010, studies had to be started rapidly across Europe in order to address this signal. Possibilities to evaluate this safety signals were limited to observational studies which, as a biological mechanism for the observed adverse event was not known, involved a great deal of guess work on risk windows, potential confounding factors and alternative explanations for observed association. As an answer on the potential association between Pandemrix and narcolepsy was needed rapidly, studies performed had limited follow-up time. They had to do with the resources available at the time which meant that they could not always rely on blinded, prospectively collected data on vaccination for the study population or on detailed prospectively collected data on potential confounding variables such as underlying comorbidities.

To add to this, suspicion of a potential association between narcolepsy and Pandemrix was already spreading amongst healthcare professionals in Finland from as early as February 2010 and was general knowledge after August 2010 when regulatory agencies published on the association which was picked up by the media. Knowledge on the association with vaccination may have resulted in a reduction of the onset-to-diagnosis interval in vaccinated individuals, whereas this would not happen to the same extent in non-vaccinated subjects. Knowledge of a putative association between vaccination and a specific event could also have resulted in patients placing symptom onset after vaccination.

The simulations described in this article illustrate that in the absence of a real association between the vaccine and narcolepsy presence of detection bias and differential exposure misclassification could account for the elevated risks detected. Moreover, the simulations also suggest that it would be too early to dismiss an elevated risk of narcolepsy following other influenza vaccines or influenza infection based upon absence of associations in observational studies alone. The veracity of the association between narcolepsy and Pandemrix will become clearer as studies are conducted with longer follow-up times, especially when studies into potential mechanisms are taken into account.

The uncertainties surrounding the role Pandemrix may have played in the surge of narcolepsy diagnoses seen in several European countries which still exist to date do underline the need to improve the infrastructures available in Europe to monitor vaccine safety and evaluate vaccine safety signals if these were to emerge. Moreover, they point toward the need to further develop methods for rapid safety assessment, such as sequential monitoring, and the need to develop methods which can adjust for stimulated diagnosis in the presence of awareness. Finally, a systematic assessment of potential sources of bias and their impact should be an integral part of any assessment of a safety signal that relies on observational studies. Without such an assessment great caution should be exerted before drawing any conclusions from these studies.

Five-year view

It is not possible to predict when and how a future pandemic will evolve. Although we might now have considerable experience to inform the safety profile of the existing influenza vaccines, with a new pandemic virus and a new mass-vaccination program we need to be prepared for the occurrence of new safety signals. The experience of narcolepsy has taught us that it is very helpful to have a good understanding not only of the epidemiology of potential adverse events in Europe but also of the diagnostic process for these events. It is necessary to know if there is potential for under diagnosis and what delays in diagnosis can be expected in different age groups and in different countries. Although impossible to pinpoint what adverse events will be of interest, considering the experience of influenza and (adjuvanted) influenza vaccines focus should be on neurological events and disorders with a potential auto-immune etiology. The inter-pandemic period should be used to collect data on diagnosis rates in different age groups, populations, countries and improve the understanding of differences between European countries in recording and diagnosing these conditions.

Key Issues

• Several observational studies have suggested a link between narcolepsy and Pandemrix, a pandemic influenza vaccine widely used in Europe during the 2009/2010 influenza A(H1N1) pandemic.

• These studies were conducted during a period of heightened awareness and, because of the urgent situation in which they were conducted, had several limitations which could result in biased estimates.

• We simulated the impact of two potential sources of bias in a cohort and case–control analysis evaluating the association between narcolepsy and exposure to a vaccine.

• The presence of detection bias and differential exposure misclassification could inflate risk estimates to the degree observed in several observational studies evaluating the safety signal.

• Case–control studies were less susceptible to the consequences of detection bias and differential exposure misclassification than the cohort study.

• Differential exposure misclassification had a more pronounced impact on risk estimates than detection bias.

• As only diagnosed subjects can be included as cases in any observational study, it is important to consider the impact of stimulated diagnosis, especially if the stimulated diagnosis is potentially differential on exposure.

• These simulations illustrate the importance of not only understanding background rates of adverse events of special interest prior to vaccination campaigns, but also having insight in the background onset-to-diagnosis interval.

Declaration of Interests

L Wijnans, C Dodd, M de Ridder, S Romio, D Weibel, J Bonhoeffer are working in a group that receive(d) funds from CDC and ECDC to study the association between pandemic influenza vaccine and narcolepsy. This work is outside the scope of the study grants and done independently. S Black is primary investigator of the US CDC funded SOMNIA study that is investigating the association between pandemic influenza vaccine and narcolepsy. S. Black is a consultant for GSK Vaccines. This work is outside the scope of the study grants and done independently. M Sturkenboom is leading a research group that occasionally conducts studies for pharmaceutical companies, none of which are related to this work. M Sturkenboom was principle investigator of a multi-country VAESCO-narcolepsy study and is co-applicant of the US CDC funded SOMNIA study that is investigating the association between pandemic influenza and narcolepsy. This work is outside the scope of the study grants and done independently. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

Acknowledgements

The authors would like to thank Dr. Hector Izurieta for his help in the conceptualization of this project.

Appendix

Figure A1. Logit-Normal distributions from which values of bias parameters were sampled for bias settings of 30, 60, and 90%.