Re: ‘Self-controlled case series design in vaccine safety: a systematic review’ – absolute and relative measures

KEYWORDS:

• self-controlled case series
• SCCS
• vaccine safety
• attributable risk
• incidence rate ratio
• relative measures
• absolute measures

1.

In an interesting, recent article by Nie et al., the authors conducted a systematic review of vaccine safety studies that used the self-controlled case-series (SCCS) study design [1]. The authors assessed methodological characteristics of the included 105 SCCS articles and concluded that case validation, validity of the assumptions, and reporting quality should be areas of focus in future SCCS studies. We noted with interest that 12% of the included studies translated the incidence rate ratio (IRR) into an absolute estimate, such as attributable risk or excess risk. Given the increase in SCCS use since the COVID-19 pandemic, we want to clarify why researchers should not calculate an absolute measure such as those attributable to results derived from this study design.

The SCCS was originally designed for vaccine safety studies and offers a way to implicitly control for time-invariant confounding by using a case only approach [2]. This is accomplished by comparing people to themselves. The SCCS method selects people who have both the exposure and the outcome and makes a within-person comparison between the outcome rate during an exposed time period (e.g. 42 days following a vaccine) and an unexposed time period. The SCCS can only estimate a relative measure, the incidence rate ratio (IRR), and answers the question ‘why now?’ [3] In contrast, cohort studies compare an exposed group and a comparator group and answer the question ‘why me?’ [3] Cohort studies can usually estimate relative and absolute measures.

SCCS studies that calculate absolute risks will typically report them as, ‘The attributable risk for the outcome was estimated as × per one million vaccinees.’ Some of the recent SCCS studies on COVID-19 vaccines cite a paper by Wilson and Hawken where the authors describe how to calculate an absolute measure from the SCCS IRR [4]. Wilson and Hawken presented the number needed to expose to cause one excess adverse event and begin by calculating the exposed attributable fraction.

Exposed attributable fraction [5]

(IRR-1)/IRR=IRD/IR₁

where IRD = incidence rate difference and IR₁ = incidence rate among the exposed.

The main concern with calculating an absolute measure from an incidence rate ratio estimated with a SCCS study is that any absolute measure derived from a SCCS is not based on a meaningful population. The SCCS case-only design only includes people with exposure and outcome. A meaningful absolute measure should be based on a population that includes both exposed and unexposed groups that may or may not experience the outcome, such as in a cohort study. The SCCS estimates the relative incidence of the outcome by comparing exposed to unexposed time periods for people with both the exposure and the outcome. Although the total number of cases included in a SCCS may be the same as the number of cases included in a cohort study conducted in the same population, there is no direct way to correctly identify the denominator needed for an absolute measure with the information from a SCCS study alone. We have observed that some SCCS studies use external estimates of the denominator – namely estimates of the incidence of the outcome in some general population to calculate an absolute measure. This is also problematic because the external denominator population may have different covariate distributions from the SCCS study population, which would introduce confounding issues that the SCCS study was designed to avoid.

We understand the desire to report absolute measures of effect, particularly within the context of vaccine safety studies where there is a potential for harm to otherwise healthy populations. Absolute measures are admittedly easier to communicate and interpret than the incidence rate ratio, which may be large even when absolute measures are small, such as when a low incidence is divided by an even lower incidence [6]. This is particularly true for vaccine safety studies where adverse events are typically rare. However, in order to correctly report an absolute measure, the denominator needs to represent the source population from which the exposed cases arose with information about exposed non-cases, unexposed cases, and unexposed non-cases. Therefore, we recommend that if an absolute measure is desired that researchers choose to complement an IRR from a nested SCCS study with a risk difference from a cohort study [7–11]. Most SCCS studies correctly report and interpret the IRR, and the SCCS is an indispensable tool for vaccine safety research, whose use should be encouraged [12]. In this way, we can continue to generate robust evidence using SCCS methods and accurately contextualize our findings.