Errors and issues in risk of all-cause and cardiac-related mortality after vaccination against COVID-19: A meta-analysis of self-controlled case series studies

KEYWORDS:

• Meta-analysis
• COVID-19
• errors
• vaccine
• risk
• safety

This article refers to:

Risk of all-cause and cardiac-related mortality after vaccination against COVID-19: A meta-analysis of self-controlled case series studies

View responses to this article:

Reconsidering the inclusion of Ladapo’s work in the meta-analysis: Validity concerns and implications

To the editor,

The safety of COVID-19 vaccination is of paramount importance as decisions will be made regarding booster vaccinations, so it was with great interest that we read the article by Marchand et al, which purports to identify cardiac mortality elevation in people receiving COVID-19 vaccination.¹ Unfortunately, there are many errors in this report which, when corrected, completely invalidate the central thesis of the paper. These issues are sufficiently severe that they warrant immediate correction. Given the fact that, if corrected, they will reverse the findings of the paper, we would recommend the editors consider retracting this publication.

Data concerns

The authors claim to present a systematic review and meta-analysis of self-controlled case series (SCCS) studies looking at cardiac risks post-vaccination for mRNA vaccines. However, there are clear errors within the data and methodology that entirely undermine this analysis.

The key issue is that the authors have extracted many numbers incorrectly from the studies that are cited. For example, they cite a SCCS paper from England, and extract the overall hazard ratio of pre-vaccine to post-vaccine outcomes as 0.99 (0.66–1.46) in their primary model (fig 2a). The correct value is 0.88 (0.75–0.95) for 12 weeks. In the 18–24 age group, which is given significant weight in the authors’ meta-analysis, the Hazard ratio is reported as 1.08 (0.83–1.42) when in the original article it is 0.91 (0.80–1.05). In fact, despite significant effort, we cannot find where the extracted figures provided by Marchand et al appear in the citation: every value supposedly extracted from this publication in Nature Communications is incorrectly reported, as well as numerous other values that the authors have extracted. These require extensive correction, as they are undeniably wrong. In addition, the authors continually refer to the extracted numbers as “hazard ratios” and in at least one occasion as “relative risk ratios”. However, the SCCS design does not produce either of these quantities, instead giving incidence rate ratios. This continued error by the authors raises the question of whether they understood the studies that they identified, a worry that is amplified by the fact that they were incapable of extracting the numbers correctly from these studies.

Marchand et al. justify the inclusion of the Ladapo paper even though it is not peer reviewed by claiming that a publication officially produced by a Department of Health in the United States meets the definition of “published and peer reviewed.” However, there is no evidence that anyone from the Florida Department of Health other than Ladapo approved the final version of the report despite Marchand et al. referring to it as Ladapo et al. Indeed, previous versions of the report, which were obtained via a freedom of information request strongly suggest that the authors of previous versions of the report did not agree with the final published version.^2,3 Importantly, the previous version of the Ladapo report acknowledges that the primary analysis did not use the correct self-controlled case series method as it did not account for multiple exposures, and that “[the primary analysis] violates the assumption that an event does not affect subsequent exposure (for mRNA vaccines), which may introduce bias upwards..” A sensitivity analysis was dutifully performed using the correct event-dependent exposures method to look at the risk following each dose. They concluded “[the association of COVID-19 vaccination and cardiac-related mortality at 28 days] was attenuated and no longer significant when applying the more appropriate event-dependent exposures model utilized for multidose vaccines. Thus, there is little suggestion of any effect immediately following vaccination..” Another unpublished version of the Ladapo report based upon using a 42-day exposure period instead of 28 similarly did not replicate the findings of the “primary analysis.” The whole purpose of sensitivity analyses is to analyze the robustness of a result under varying circumstances, and the failure of sensitivity analysis to replicate the findings extracted by Marchand et al is quite concerning, especially given the very high weighting (92.2%, 98.3%, 99.3% in figure 3A) given the Florida SCSS in the authors metanalysis.

Methodological concerns

A meta-analysis can synthesize important information to improve power to detect signals, or to address controversies within conflicting studies.⁴ The choice of design is crucial, to minimize bias and to ensure that the results derived from the meta-analysis reflect reality and deepen our understanding of the issue.

The authors methodology states: “We analyzed homogeneous data under the fixed-effects model and heterogeneous data under the random-effects model. We assessed the heterogeneity among studies using the I2 and the p-value of the Chi-square tests. Values of P < .1 or I2 > 50% were significant indicators of the presence of heterogeneity..” However, in model 1.2.1, despite an I^2 value of 61%, the authors have performed a fixed-effects analysis. The correct analysis according to their own stated methodology would be a random-effects meta-analysis. Had the authors followed their own methodology in this case, the result would have been an aggregate IRR of 1.03 (0.91–1.16). This result entirely overturns the results and conclusion that the authors have come to, even accepting their aforementioned data extraction errors.

More broadly, the statistical choices in the manuscript are not justifiable. As is obvious from the models in figures of the study, a fixed-effects model in this case results in a single study being drastically over-weighted in the meta-analytic model. In one case, the Ladapo et al publication has > 99% weight, which makes this less a “meta”-analysis and more a recitation of this single study’s results. As described by the Cochrane collaboration, statistical heterogeneity is not a fixed quantity, and low rates of heterogeneity are not necessarily sufficient to perform a fixed-effects analysis. As is obvious from the weighting, these analyses should have all been random-effects. From a theoretical perspective, this is also clear – there is no epidemiological argument that the between-study effects for SCCS analyses conducted at different times in different countries could ever possibly be considered fixed. The England study, for example, is of young people aged 12–29, whereas the Florida study is all residents 18 and older.⁵ This directly violates a Cochrane meta-analytic principle, in that included studies should be sufficiently homogeneous in terms of participants, interventions and outcomes to provide a meaningful summary.

Finally, the use of a leave-one-out method is important, however with three studies there should be a supplemental file or table which shows the results if each of the studies is excluded, as is standard for the practice. We strongly suspect that with the proper extraction of the England study data, it would likely be that removing the Florida SCCS is as advantageous for “reducing heterogeneity” as is removing the Italian SCCS.

With significant methodological issues, important omissions about one of the studies, and errors in data extraction, we conclude that the central findings of the paper are very unlikely, and in fact, could be the reverse. We recommend immediate correction for all identified issues, and given the conclusion of the article could be incorrect, a retraction is likely in order.

With respect,

Tyler Black, University of British Columbia, Vancouver, BC, Canada

Gideon Meyerowitz-Katz, University of Wollongong, Wollongong, NSW, Australia

Susan Oliver, University of New South Wales, Sydney, NSW, Australia

Nana Owusu-Boaitey, University of Maryland, College Park, Maryland, USA

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was not supported by any funding.