https://orcid.org/0000-0002-0387-0576
?Mathematical formulae have been encoded as MathML and are displayed in this HTML version using MathJax in order to improve their display. Uncheck the box to turn MathJax off. This feature requires Javascript. Click on a formula to zoom.ABSTRACT
Although nationwide immunization with SARS-CoV-2 mRNA vaccines began in February 2021, the evaluation of vaccine effectiveness (VE) using a test-negative design has not been conducted adequately in Japan. To evaluate the effectiveness of the SARS-CoV-2 mRNA vaccines, we conducted a test-negative case–control study during the periods dominated by the Delta and Omicron variants. In total, 518 and 358 adult participants with COVID-19-like symptoms were tested for the virus from August to October 2021 (Delta variant predominance) and in February 2022 (Omicron variant surge), at the Kawasaki Saiwai Clinic. During Delta variant predominance, the effectiveness of full vaccination was 90.4% (95% confidence interval [CI]: 82.1–94.8) and 97.3% (95% CI: 71.7–99.7) against all COVID-19 and moderate-to-severe disease, respectively. However, partial vaccination failed to show effectiveness against moderate-to-severe COVID-19. The effectiveness of the mRNA vaccines against all COVID-19 infection declined to 16.1% (95% CI: −81.0 to 61.1) in February 2022. Our results indicated that, although mRNA vaccines showed significant preventive effects against all COVID-19 during Delta variant predominance, these preventive effects waned during Omicron variant surge. To the best of our knowledge, this is the first study that evaluated VE in the Japanese population during both periods.
Introduction
In Japan, the National Immunization Program (NIP) against SARS-CoV-2 began in February 2021. At first, essential health-care workers and workers who are essential to maintain critical infrastructure and continue critical services and functions were involved. After that NIP included the general population by the end of May 2021. Approval of the first mRNA vaccine, BTN162b2 (Comirnaty; Pfizer-BioNTech) was followed by that of a second mRNA vaccine, mRNA-1273 (Spikebacks; Moderna). A placebo-controlled clinical trial was used to analyze the vaccine efficacy (VE) and safety of both these vaccines, which had been approved for Emergency Use Authorization (EUA) by the Food and Drug Administration (FDA).Citation1–6 Following authentication, the number of studies investigating VE, safety in a real-world setting, or immunogenicity induced by boosters, have been increasing.Citation7–10 Although studies that use the test-negative design to evaluate VE in a real-world setting are increasing, reports on long-term VE evaluation in Japan are yet to be published.Citation11–18
Case–control studies involving the test-negative case–control design are frequently used to evaluate the VE of seasonal influenza vaccines every year. In that design, VE is represented by the difference between vaccination rates of test-positive cases and test-negative controls.
To analyze the VE of mRNA vaccines designed against SARS-CoV-2 for the period during which the Delta variant became predominant and then waned, as well as for the period during which the Omicron variant surged, we used real-world evidence from our clinic to conduct a test-negative case–control study.
Materials and methods
The first group of participants were adults who visited our clinic because of at least one COVID-19-like symptom or recent history of close contact with COVID-19 patients, and underwent a virus detection test via a polymerase-chain-reaction (PCR), or a nicking-enzyme-amplification-reaction (NEAR) or an antigen test for SARS-CoV-2 at the Kawasaki Saiwai Clinic from August to October 2021, the period of Delta variant predominance. The second group of adults who presented under circumstances that were similar to the first group were investigated in February 2022, the period of Omicron variant surge. The Kawasaki Saiwai Clinic is a medium-sized urban facility located in the central area of Kawasaki city, the sixth largest city in Japan. The clinic has 22 subspecialty ambulatory departments but no ward. COVID-19-like symptoms were defined as follows: (i) fever, (ii) acute cough, (iii) acute-onset dyspnea, (iv) fatigue, or (v) loss of taste/smell. Close contact with COVID-19 patients was based on the criteria of the Japanese Ministry of Health, Labor and Welfare as followings, a face-to-face exposure to the COVID-19 patients within the distance of 1 m or shorter without a mask for more than 15 min.Citation19
Percutaneous oxygen saturation (SpO2), virus detection test results, vaccination history, and vaccine type were evaluated using electrical medical records retrospectively. Cases and controls were defined on the basis of a positive virus detection test and a negative virus detection test, respectively. If either PCR, NEAR, or antigen test was positive, persons were judged as test-positive in both groups. NEAR (ID NOW™) or a rapid antigen test performed in-house was approved by the Japanese Ministry of Health, Labor and Welfare. In clinical practice, ID NOW™ was performed for individuals who required urgent medical intervention especially because of acute conditions, such as respiratory distress or others. ID NOW™, which uses the NEAR method, can provide results within 13 min.Citation20 However, because of the lower sensitivity of ID NOWTH (approximately 70%) and antigen test, for the persons who presented COVID-19-like symptoms with negative these rapid tests, we added PCR to confirm test-negative in both groups. During the period of Omicron variant surged, the Japanese Ministry of Health, Labor and Welfare recommended that COVID-19 could be diagnosis based on the positive results of a self-test using a private brand antigen detecting kit. We added PCR for the persons with COVID-19-like symptoms who were negative for these rapid tests. Vaccination status was defined as follows: partial vaccination (test performed 14 d after receipt of the first dose through 6 d after receipt of the second dose); full vaccination (test performed ≥7 d after receipt of the second dose); or unvaccinated (test performed <14 d after receipt of the first dose or never vaccinated). Moderate-to-severe disease was defined as a SpO2 less than 96% based on the criteria of Japanese Ministry of Health, Labor and Welfare.Citation19 VE of partial and full vaccination for preventing symptomatic disease and severe disease was evaluated. The suppressive effect of positive conversion of virus detection tests in people who had close contact with COVID-19 patients, regardless of with or without COVID-19-like symptoms was also evaluated.
Vaccine effectiveness (VE) was calculated as follows:
The age- and gender-adjusted VE for preventing all COVID-19 infection was calculated for all mRNA vaccines using the logistic regression model. The age was used as a binary variable (e.g. <65 y and ≥65 y). To make comparisons between categorical variables or continuous variables, Fisher’s exact tests or Mann–Whitney U test were used, respectively. All statistical analyses were performed via EZR (Saitama Medical Center, Jichi Medical University, Saitama, Japan), a graphical user interface for R (The R foundation for Statistical Computing, Vienna, Austria). More precisely, it is a modified version of R commander designed to add statistical functions frequently used in biostatistics.Citation21 The study protocol was approved by the Ethical Committee of the Kawasaki Saiwai Clinic (No. 2021–01), and the study was conducted in accordance with the principles of the Declaration of Helsinki.
The sample size corresponding to an α error of 5% and a detection power of 80% was estimated. We assumed a VE of at least 75% based on a previous report,Citation14 a test-positive rate of 10% observed in our facility during the previous year (e.g., assumed case–control ratio of 1:9), and a vaccination rate of 50% observed in Kawasaki City during the study period. A sample size of 216 (e.g., 22 cases and 194 controls) was required to detect a possible 75% VE for preventing the onset of COVID-19 symptoms during the period of Delta variant predominance. We also assumed a VE of 50% against the Omicron variant, based on the report by the National Institute of Infectious Diseases,Citation22 a test-positive rate of 50% observed in our facility during the previous month, and a vaccination rate of 80% as observed in Kawasaki City during that period. A sample size of 343 (e.g., 172 cases and 172 controls) was required to detect a possible VE of 50% for preventing the onset of COVID-19 symptoms during the period of Omicron variant surge.
Results
The characteristics of the participants are shown in . We analyzed 518 adults during the period of Delta variant predominance and 358 during the period of Omicron variant surge. Although median age, gender distribution, and SpO2 were almost identical, the test-positive rate and the unvaccinated rate were 32.6% and 53.2%, respectively, during the Delta variant predominant period, and almost 70% and 9.5%, respectively, during the Omicron variant surge. During the period of Delta variant predominance, the test-negative participants were older (44 vs 42 y; P = .021) and contained more females (51% vs 39.1%, P = .011) than the test-positive participants. On the contrary, the persons whose SpO2 less than 96% and the unvaccinated persons were more frequent in the test-positive participants. However, no differences were observed between the SpO2 of participants. No difference between the ages, gender distributions, or SpO2 of the test-negative and -positive participants was observed during the period of the Omicron variant surge.
Table 1. Characteristics of participants during the periods of Delta variant predominance and Omicron variant surge.
The VE during the period of Delta variant predominance is shown in . The VE of full and partial vaccination with BTN162b2 for preventing all COVID-19 infections was 89.6% (95% confidence interval [CI]: 77.8–95.0) and 62.1% (95% CI: 17.4–82.5), respectively. The VE of full and partial vaccination with mRNA-1273 for preventing all COVID-19 infection was 89.0% (95% CI: 71.3–95.8) and 90.6% (95% CI: 25.9–98.8), respectively. The VE for sum of full and partial vaccination with any mRNA vaccines, for preventing all COVID-19 infection was 90.4% (95% CI: 82.1–94.8) and 75.0% (95% CI: 50.5–87.4), respectively. Significant preventive effects were observed for both full and partial vaccination. The participants were divided into two age groups, under 65 y and 65 y or more, and their VE evaluated. The VE for preventing all COVID-19 infection was 87.7% (95% CI: 55.2-96.6) and 91.2% (95% CI: 81.7–95.7) for the group under 65 y and the one representing 65 y or more, respectively.
Table 2. Vaccine effectiveness of the mRNA vaccines against SARS-CoV-2 during the period of Delta variant predominance.
The VE for full vaccination with BTN162b2 or a sum of any mRNA vaccines for preventing moderate-to-severe COVID-19 disease was 99.9% (95% CI: 74.2–99.9) and 97.3% (95% CI: 71.7–99.7), respectively. However, participants whose SpO2 was less than 96% were very few; they were especially found among partially vaccinated persons.
For the 95 persons with a recent history of close contact with COVID-19 patients, the VE for preventing positive conversion was 90.6% (95% CI: 74.0–96.6).
The VE during the period of Omicron variant surge is shown in . The analysis of 358 persons during the period of Omicron variant surge did not indicate a significant preventive effect. The persons who had a history of COVID-19 in our subjects were less than ten in the second group. The VE for sum of any mRNA vaccines for preventing symptomatic COVID-19 disease was 16.1% (95% CI: −81.0 to 61.1).
Table 3. Vaccine effectiveness of the mRNA vaccines against SARS-CoV-2 during the period of Omicron variant surge.
Discussion
Our study involving a test-negative case–control design in a real-world setting showed that crude as well as adjusted VE of both mRNA vaccines showed an effectiveness of approximately 90% for preventing all COVID-19 infection only for the Delta period.Citation11–18 Except for one report, which evaluated VE against the Delta variant in the Japanese general population, there have been no published reports evaluating long-term VE in Japan.Citation18 Thus, to the best of our knowledge, this study of ours is the first to evaluate VE against both Delta and Omicron variants. The test-negative design is based on the selection of study participants presenting at medical institutions with symptoms, such as fever or acute respiratory symptoms. Therefore, the designs of such studies are limited by conditions that determine whether or not a person seeks medical attention due to the illness, and thus differs from placebo-controlled clinical trials concerning both mRNA vaccines for EUA from the FDA.Citation1–10 However, the VE for preventing all COVID-19 onsets for both mRNA vaccines, which were determined by test-negative designs, was almost identical to that of previous controlled clinical trials.
Two doses of BNT162b2 as well as two doses of any mRNA vaccines showed significant preventive effect against moderate-to-severe COVID-19 disease among persons less than 65 y. However, very few individuals had moderate-to-severe COVID-19. Although VE was not very robust, these results may imply that meaningful VE of mRNA vaccines to prevent moderate-to-severe COVID-19 was shown. In contrast, the sum of any mRNA vaccines failed to show an acceptable VE against moderate-to-severe COVID-19 disease, where the majority of participants had mild disease, with only a few cases presenting with moderate-to-severe disease. Thus, our study may have been too underpowered to detect the VE for preventing moderate-to-severe disease, especially in younger aged persons.
During the period of Omicron variant surge, the VE of the sum of any mRNA vaccines for preventing symptomatic COVID-19 disease was not significant. The true VE for this period may not have been reflected in our results due to declining vaccine titers, reduced immunogenicity, or a low number of participants. A waning of VE was observed by us during this period, which was substantiated by several other studies that showed a waning of VE, 5 months or more after second dose vaccination.Citation11,Citation13,Citation15 Furthermore, rapidly declining antibody titers have also been reported by other studies.Citation23 Japanese NIP initiated for general population at the end of May; hence, in our study, at least one dose vaccination rate was 46.7% during Delta variant predominance. The time from vaccination was within 4 months or shorter in our first group. In contrast, two doses vaccination rate during Omicron variant surge was 90.5% in our second group. Because of the difference in vaccination rates between both periods, in the half of our second group, the time from vaccination may be assumed as 4 months or shorter in the period of Omicron variant surge. The vaccination status was judged based on the same criteria shown in Materials and Methods during both periods. Hence, direct evidence of declined long-term VE in the second group was not assessed. Although the immunogenicity to the Omicron variant following two mRNA vaccine doses has not been reported, reduced immunogenicity was assumed due to observed mutation of the targeted epitope. Because the time from vaccination was not fully assessed during Omicron surge, not only long-time passage from vaccination but also lower immunogenicity against Omicron variant may have contributed to the low VE of mRNA vaccines in this period. Although the persons who had a history of COVID-19 were less than ten in second group, the possibilities that these persons may have led to an underestimate of the infection rate and influenced the VE were not excluded. The actual test-positive rate (70%) and the vaccination rate (90%) during this period were higher than the rates assumed by us for sample size estimation, and therefore more participants may have been needed to obtain a more accurate estimate of the VE. Notably, lower specificity of laboratory tests may also have contributed to the underestimation of VE. In order to compensate for the lack of social resources of PCR tests that existed during the period of Omicron variant surge, the Japanese Ministry of Health, Labor and Welfare stipulated that COVID-19 may be diagnosed based on private rapid antigen tests for SARS-CoV-2 detection, instead of authentic PCRs. Therefore, the effect exerted by inaccurate test-positive cases diagnosed using positive rapid antigen tests on our study cannot be excluded, because 50 test-positive cases were based on these positive rapid antigen tests. Thus, the lower specificity of rapid antigen tests may have led to an underestimation of VE.
Several limitations deemed to have affected this study. First, selection bias cannot be excluded. The participants of this study may not represent the entire COVID-19 population in Japan, because the majority of our participants were mild cases presenting in a single center ambulatory setting. Second, the time from vaccination was not fully assessed during the period of Omicron variant surge; hence, direct evidence of declined long-term VE in the second group was not assessed. Third, lower specificity of the self-test kits adopted for the detection of SARS-CoV-2 during the period of Omicron variant surge may have influenced the VE because of the nature of the test-negative design. Fourth, genotyping of the virus was not performed for the test-positive cases, resulting in the VE not reflecting a direct preventive effect against either the Delta variant or the Omicron variant during each period. Fifth, there deemed to be several other confounders, such as comorbidities and previous history of COVID-19 in the participants. There were incomplete descriptions concerning the comorbidities because of the retrospective review of medical records. Hence, the comorbidities of the participants were not adopted for the analysis. Also, the possibilities that previous history of COVID-19 influenced the VE could not be excluded. Despite the abovementioned limitations, our results have shown that BTN162b2 and mRNA-1273 are capable of exerting the expected preventive effect on the all COVID-19 infection besides moderate-to-severe disease only during the period of Delta variant predominance, and also demonstrated that these preventive effects on the Japanese population were susceptible to waning during the period of Omicron variant surge. To the best of our knowledge, this report is the first to evaluate the VE during the both periods of Delta variant predominance and Omicron variant surg using the test-negative design; this method is considered to be as accurate as the randomized control trial (RCT) method, and therefore useful for understanding VE during a pandemic situation.
Acknowledgements
We would like to thank Editage (www.editage.com) for English language editing.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Additional information
Funding
References
- Anderson EJ, Rouphael NG, Widge AT, Jackson LA, Roberts PC, Makhene M, Chappell JD, Denison MR, Stevens LJ, Pruijssers AJ, et al. Safety and immunogenicity of SARS-CoV-2 mRNA-1273 vaccine in older adults. N Engl J Med. 2020;383(25):2427–6. PMID: 32991794. doi:10.1056/NEJMoa2028436.
- Jackson LA, Anderson EJ, Rouphael NG, Roberts PC, Makhene M, Coler RN, McCullough MP, Chappell JD, Denison MR, Stevens LJ, et al. An mRNA vaccine against SARS-CoV-2 — preliminary report. N Engl J Med. 2020;383(20):1920–31. PMID: 32663912. doi:10.1056/NEJMoa2022483.
- Polack FP, Thomas SJ, Kitchin N, Absalon J, Gurtman A, Lockhart S, Perez JL, Marc GP, Moreira ED, Zerbini C, et al. Safety and efficacy of the BNT162b2 mRNA Covid-19 vaccine. N Engl J Med. 2020;383(27):2603–15. PMID: 33301246. doi:10.1056/NEJMoa2034577.
- Walsh EE, Frenck RW Jr, Falsey AR, Doblecki-Lewis S, Martin JM, Anderson EJ, Campbell TB, Clark J, Jackson LA, Fichtenbaum CJ, et al. Safety and immunogenicity of two RNA-based Covid-19 vaccine candidates. N Engl J Med. 2020;383(25):2439–50. PMID: 34551225. doi:10.1056/NEJMoa2113017.
- Baden LR, El Sahly HM, Essink B, Kotloff K, Frey S, Novak R, Diemert D, Spector SA, Rouphael N, Creech CB, et al. Efficacy and safety of the mRNA-1273 SARS-CoV-2 vaccine. N Engl J Med. 2021;384(5):403–16. PMID: 33378609. doi:10.1056/NEJMoa2035389.
- El Sahly HM, Baden LR, Essink B, Doblecki-Lewis S, Martin JM, Anderson EJ, Campbell TB, Clark J, Jackson LA, Fichtenbaum CJ, et al. Efficacy of the mRNA-1273 SARS-CoV-2 vaccine at completion of blinded phase. N Engl J Med. 2021;385(19):1774–85. PMID: 34551225. doi:10.1056/NEJMoa2113017.
- Sharif N, Alzahrani KJ, Ahmed SN, Dey SK. Efficacy, immunogenicity and safety of COVID-19 vaccines: a systematic review and meta-analysis. Front Immunol. 2021;12:714170. PMID: 34707602. doi:10.3389/fimmu.2021.714170.
- Atmar RL, Lyke KE, Deming ME, Jackson LA, Branche AR, El Sahly HM, Rostad CA, Martin JM, Johnston C, Ruppet RE, et al. Homologous and heterologous Covid-19 booster vaccinations. N Engl J Med. 2022;386(11):1046–57. PMID: 35081293. doi:10.1056/NEJMoa2116414.
- Chu L, Vrbicky K, Montefiori D, Huang W, Nestorova B, Chang Y, Carfi A, Edwards DK, Oestreicher J, Legault H, et al. Immune response to SARS-CoV-2 after a booster of mRNA-1273: an open-label phase 2 trial. Nature Med. 2022;28(5):1042–49. PMID: 35241844. doi:10.1038/s41591-022-01739-w.
- Poh XY, Tan CW, Lee IR, Chavatte J, Fong S, Prince T, Hartley C, Yeoh AY, Rao S, Chia PY, et al. Antibody response of heterologous vs homologous mRNA vaccine boosters against the SARS-CoV-2 Omicron variant: interim results from the PRIBIVAC study: a randomized clinical trial. Clin Infect Dis. 2022;ciac345. PMID: 35543372. doi:10.1093/cid/ciac345.
- Andrews N, Stowe J, Kirsebom F, Toffa S, Rickeard T, Gallagher E, Gower C, Kall M, Groves N, O’Connell A, et al. Covid-19 vaccine effectiveness against the Omicron (B.1.1.529) variant. N Engl J Med. 2022;386(16):1532–46. PMID: 35249272. doi:10.1056/NEJMoa2119451.
- Pilishvili T, Gierke R, Fleming-Dutra KE, Farrar JL, Mohr NM, Talan DA, Krishnadasan A, Harland KK, Smithline HA, Houet PC, et al. Effectiveness of mRNA Covid-19 vaccine among U.S. Health Care Personnel. N Engl J Med. 2021;385(25):e90. PMID: 34551224. doi:10.1056/NEJMoa2106599.
- Chemaitelly H, Tang P, Hasan MR, AlMukdad S, Yassine HM, Benslimane FM, Al Khatib HA, Coyle P, Ayoub HH, Al Kanaani Z, et al. Waning of BNT162b2 vaccine protection against SARS-CoV-2 infection in Qatar. N Engl J Med. 2021;385(24):e83. PMID: 34614327. doi:10.1056/NEJMoa2114114.
- Tang P, Hasan MR, Chemaitelly H, Yassine HM, Benslimane FM, Al Khatib HA, Mukdad Sawsan A, Coyle P, Ayoub HH, Al Kanaani Z, et al. Bnt162b2 and mRNA-1273 COVID-19 vaccine effectiveness against the SARS-CoV-2 Delta variant in Qatar. Nature Med. 2021;27(12):2136–43. PMID: 34728831. doi:10.1038/s41591-021-01583-4.
- Price AM, Olson SM, Newhams MM, Halasa NB, Boom JA, Sahni LC, Pannaraj PS, Irby K, Bline KE, Maddux AB, et al. Bnt162b2 protection against the Omicron variant in children and adolescents. N Engl J Med. 2022;386(20):1899–909. PMID: 35353976. doi:10.1056/NEJMoa2202826.
- Accorsi EK, Britton A, Fleming-Dutra KE, Smith ZR, Shang N, Derado G, Miller J, Schrag SJ, Verani JR. Association between 3 doses of mRNA COVID-19 vaccine and symptomatic infection caused by the SARS-CoV-2 Omicron and Delta variants. JAMA. 2022;327(7):639–51. PMID: 35060999. doi:10.1001/jama.2022.0470.
- Andrews N, Tessier E, Stowe J, Gower C, Kirsebom F, Simmons R, Gallagher E, Thelwall S, Groves N, Dabrera G, et al. Duration of protection against mild and severe disease by Covid-19 vaccines. N Engl J Med. 2022;386(4):340–50. PMID: 35021002. doi:10.1056/NEJMoa2115481.
- Hara M, Furue T, Fukuoka M, Iwanaga K, Matsuishi E, Miike T, Sakamoto Y, Mukai N, Kinugasa Y, Shigyo M, et al. Real-World effectiveness of the mRNA COVID-19 vaccines in Japan: a case–control study. Vaccines. 2022;10(5):779. PMID: 35632535. doi:10.3390/vaccines10050779.
- Japanese Ministry of Health, Labor and welfare. Shingata CORONA VIRUS Kansensyou COVID-19 Shinryo no Tebiki Dai 7.2 Han [Our environment: the practice guideline for novel Corona virus infection COVID-19 ver 7.2]. Tokyo (Japan) Affairs; 2022 May 9 [ accessed 2022 May 10]. https://www.mhlw.go.jp/content/000967699.pdf
- Yamamoto K, Ohmagari N. Microbiological testing for coronavirus disease 2019. JMA J. 2021;4:67–75. PMID: 33997438. doi:10.31662/jmaj.2021-0012.
- Kanda Y. Investigation of the freely available easy-to-use software ‘EZR’ for medical statistics. Bone Marrow Transplant. 2013;48(3):452–58. PMID: 23208313. doi:10.1038/bmt.2012.244.
- Arashiro T, Arima Y, Muraoka H, Sato A, Oba K, Uehara Y, Arioka H, Yanai H, Huramochi JKY, Genei Ihara G, et al. COVID-19 vaccine effectiveness against symptomatic SARS-CoV-2 infection during Delta-dominant and Omicron-dominant periods in Japan: a multi-center prospective case-control study (FASCINATE study). Clin Infect Dis. 2022;ciac635. doi:10.1093/cid/ciac635.
- The National Institute of Infectious Diseases. Omicron Kabu Ryukouki ni okeru Shingata CORONA Vaccine no Yukosei [Our environment: Vaccine effectiveness of novel Corona vaccine during Omicron surge-Interim report] Tokyo (Japan) [ accessed 2022 Feb 2]. https://www.niid.go.jp/niid/ja/2019-ncov/2484-idsc/10966-covid19-71.html.