ABSTRACT
Background: The test-negative design has been used widely in evaluation of various vaccines’ effectiveness, such as influenza, rotavirus, and so on. Recently, there have been some studies about EV-71 vaccine effectiveness by using test-negative design(TND). However, the validity of the TND application in EV-71 vaccines has not been evaluated.
Methods: This study is set upon prior methods to evaluate the validity of TND for influenza vaccine by using a randomized controlled clinical trial database. Vaccine effectiveness estimated by TND (VE-TND) in modified intention-to-treat population (mITT) and per-protocol-set population(PPS) was derived from a large randomized placebo-controlled clinical trial (RCT) of inactivated monovalent EV-71 vaccine in China. Derived VE-TND estimates were compared to the original vaccine efficacy results in RCT (VE-RCT).
Results: We totally enrolled 7325 participants who seeked medical care for suspected EV-71 infected diseases during the surveillance. There are no significant differences between cases(test-positive) and controls(test-negative) on sex, age, height, and weight. TND vaccine effectiveness estimates were similar to original RCT vaccine efficacy estimates, both in modified intention-to-treat population and per-protocol populations.
Conclusions: This study supports that TND, as an appropriate observational study design is valid to measure EV-71 vaccine effectiveness.
1 Introduction
EV-71 infection could cause various diseases including hand, foot, and mouth disease (HFMD), herpangina, neurological signs and nonspecific illnesses, affecting mainly children younger than 5 years-old.Citation1 EV-71 has emerged as a major neurotropic virus responsible for severe neurological complications and has become a serious public health threat to infants and young children across the Asia-Pacific region. Since 1990s, EV-71 infections have become the primary cause of HFMDs. In China, more than 10 million cases of HFMD with more than 3000 fatalities were reported between 2008 and 2012.Citation2,Citation3 Currently, there is no approved antiviral drug against EV-71 infection.Citation4 An efficacious prophylactic vaccine is the best choice to prevent EV-A71 infection and disease. Until 2016, three inactivated monovalent EV-71 vaccines (developed by Chinese Academy of Medical Sciences (CAMS), Sinovac Biotech, and Beijing Vigoo Biological) were licensed to prevent EV-71 infection in China.Citation5 These EV-71 vaccines showed high efficacy against EV71-associated diseases in phase III randomized controlled trials.Citation6–8 The vaccines’ efficacy rates were 97.4%(CAMS), 94.8%(Sinovac) and 90 · 0%(Beijing Vigoo), separately.
However, the studies about evaluating the effectiveness of EV-71 vaccine after licensure are only few. So far, three studies have been published about evaluating the effectiveness of EV-71 vaccine in the real-world.Citation5,Citation9,Citation10 The EV71 vaccine effectiveness rates were 85 · 4%(Henan), 88.3%(Guangxi) and 83.7%(Beijing), separately. They all used “test-negative design”(TND) to estimate the effectiveness of EV-71 vaccine. They did a pioneering work and gave a template for evaluating the effectiveness of EV-71 vaccines after licensure by TND.Citation11
TND is considered as a variant of the traditional case-control study and has become popular for post-licensure observational studies of the effectiveness of vaccines, especially for influenza and rotavirus vaccines.Citation12,Citation13 In this design, patients seeking medical care for a predefined clinical condition are tested for a specific viral infection by using a highly sensitive and specific laboratory test. Those tested positive are cases and controls are tested negative for infection but meeting the same enrollment criteria. Vaccine status is compared between test-positive cases versus test-negative controls.Citation14
The real-world EV-71 vaccine effectiveness estimates showed that the effectiveness of EV-71 vaccines are a little lower than the efficacy in phase III randomized controlled trials.Citation5,Citation9,Citation10 Maybe it reflected the difference of vaccine’s effectiveness between the real world and clinical-trial situation. Or possibly, the accuracy and precision of TND for EV-71 vaccines evaluation is not steady enough. Therefore, we would like to examine the validity about the TND application in EV-71 vaccines. De Serres, G. etc used databases from four influenza vaccine randomized controlled trials(RCT) to verify the validity about the TND application in influenza vaccines.Citation15 The RCTs represent the ideal background to assess the validity of the TND,Citation15 providing the comparability of vaccinated and unvaccinated participants with follow-up and disease ascertainment which could minimize the bias and confounding. So here we did a post-hoc analysis and try to confirm the validity of TND for EV-71 vaccine by using the database from a randomized controlled trial of EV-71 vaccine.
2 Method
2.1 Data source
The database is from a multi-center, randomized, double-blind, placebo-controlled, phase 3 trial, which conducted in China(NCT01508247).Citation8 The first EV71 vaccine phase 3 trial performed by Vigoo had high levels of quality control and data. It could provide the ideal setting for us to simulate the TND approach The inactivated alum-adjuvant EV-71 vaccine was developed by Beijing Vigoo Biological, which children aged 6–35 months received at 0 and 28 days. The trial enrolled 10245 children and infants, 5120 participants in vaccine group and 5125 in placebo group. Details can be found in the related publication.Citation8 In original phase 3 trial, we did a high-quality combination of active and passive monitoring to capture any suspected EV71 cases. Thus, we concluded that we had no omission of potential cases and this was why phase 3 trial could provide an ideal setting for us to simulate TND approach. Participants were actively followed after being administered two doses of vaccine or placebo. Guardians were encouraged to take their children to treatment in clinics or hospitals for any illness. The surveillance was from 56 days after first dose to 14 months. We collected and sort out information from those participants who had suspected illness associated EV71 infection and did a test-negative design to evaluate the EV-71 vaccine effectiveness for Enterovirus 71 (EV71)-associated diseases, EV71-associated HFMD, and EV71-associated other cases.
2.2 Case definitions and laboratory testing
EV71-associated disease is defined as clinical symptoms including HFMD, herpangina, neurological signs (aseptic meningitis or encephalitis) with or without serious sequelae, and nonspecific illnesses – e.g., febrile illness, viral exanthema, respiratory infection.Citation16 HFMD-like illness is characterized by febrile illness with papulovesicular rash (can occasionally be maculopapular without vesicular lesion) on palms and soles, with or without vesicles or ulcers in the mouth, buttocks, knees, or elbows. EV71-associated other cases are EV71-associated disease except HFMD.Citation8
All potential cases were reported by the clinics or hospitals. Throat or rectal swabs or both were taken for pathogen detection within 24 h. These samples were tested for EV71 by real-time PCR (fluorescence assay) with a viral RNA diagnostic kit. A participant was defined as a case if at least one positive test result during follow-up. A participant was defined as a control if at least one negative test results during follow-up and the participant had no positive results.
2.3 Statistical analysis
The analysis was based on the mITT population and PPS populations so that we could compare the results with RCT’s. The modified intention-to-treat population included those participants who received at least one dose. The per-protocol population included those participants who received two doses. TND vaccine effectiveness (VE-TND) was defined as (1- Odds Ratio) *100. Logistic regression was used to estimate the odds ratio and associated 95% confidence intervals (CI) of test-positive participants compared to test-negative participants by vaccine and placebo status. We used the χ2 test or Fisher’s exact test to analyze categorical data, ANOVA to analyze continuous data. Statistical analyses were done with SAS software (version 9.4). All statistical tests were two-sided, and statistical significance was defined as P < .05.
3 Results
3.1 Participant characteristics
During the surveillance, we totally enrolled 7325 participants who seeked medical care for suspected EV-71-infected diseases, 52 tested positive as cases and 7273 tested negative as controls in modified intention-to-treat population, 49 tested positive as cases and 7088 tested negative as controls in per-protocol population. shows the characteristics between cases and controls. There are no significant statistical differences between cases and controls on sex, age, height, and weight. Baseline characteristics of groups were balanced with so that further adjustment for residual confounding was not required. shows the major EV-71 infection epidemic curves. EV71 infection major occurred from April to May and from November to December. Our surveillance embraced two epidemic period about EV71.
Table 1. Comparison of demographic characteristics between cases and controls in mITT and PPS
3.2 Vaccine effectiveness
There were 1453 participants had HFMD-like illness, 36 tested positive while 1417 tested negative. The rest were 5872 EV71-associated other cases, 16 tested positive and 5856 tested negative. shows the test-negative vaccine effectiveness and the original RCT vaccine efficacy. The VE-TND against EV71-associated disease, EV71-associated HFMD and EV71-associated other cases are 81.7%(95%CI: 61.6,91.4), 91.2%(95%CI: 71.2,97.3), and 53.9%(95%CI: −32.9,84.0), separately.
Table 2. VE-TND and VE-RCT results
In the per-protocol population, 1424 participants seeked medical care for HFMD-like illness and 33 of them tested positive while 1391 tested negative. There were 5713 EV71-associated other cases, 16 of them tested positive and 5697 tested negative. The VE-TND against EV71-associated disease, EV71-associated HFMD and EV71-associated other cases are 80.2(95%CI: 57.8,90.8), 90.3(95%CI: 68.0,97.0), and 53.5(95%CI: −33.9,83.9), separately. summarizes the comparison of estimates and 95% CIs between VE-TND and VE-RCT.
Figure 1. VE-TND and VE-RCT and 95% confidence intervals in mITT
Figure 2. VE-TND and VE-RCT and 95% confidence intervals in PPS
Figure 3. Suspected EV71 infected cases epidemic curves in mITT and PPS
4 Discussion
Overall, the results from the TND analysis for EV-71 vaccine effectiveness were similar to the original RCT efficacy results, both in modified intention-to-treat population and per-protocol population. Our research result is similar to the previous study in influenza and rotavirus vaccine.Citation15,Citation17 G De Serres etc and Lauren M. Schwartz etc both used per-protocol population data to calculate the VE-TND and compared them with the results of corresponding RCTs. Their estimates of TND were not meaningfully different from efficacy results. Not only we received the same result in per-protocol population database, but also in modified intention-to-treat population. Thus, the TND method could be applied widely in evaluating effectiveness of EV-71 vaccine furthermore.
We reviewed three-phase 3 trials and found that the definitions about EV71 disease were similar because they all made a comprehensive definition about EV71 disease and HFMD. But in three TND studies, the enrolled criteria they used were different. In Henan’s study, they enrolled severe HFMD inpatients aged 6–71 months with comprehensive definition about EV71 disease. In Beijing’s study, they enrolled all suspected HFMD outpatients aged 6–60 months while in Guangxi’s study severe HFMD cases under 12 years old were enrolled. We suppose comprehensive definition about EV71 disease and HFMD should be more suitable because EV71 could cause various diseases more than HFMD. And, although most HFMD cases may have typical clinical symptoms, there are still some special and untypical cases so detailed and completed definition is needed. As Sullivan, S. G.’s reviews mentioned,Citation18 in influenza TND research, definition of disease may affect positive and negative ratio of test results and then affect the estimated VE. Then biases could persist under the comprehensive definition.Citation19 Thus, in EV71 TND studies, definition of disease needs to be further considered. The importance of clear and specific case definitions should not be overlooked.
Except the definition of disease, the severity of disease and study subjects(such as inpatient or outpatient) are also potential bias to VE.Citation19 Disease outcomes may consist of a broader spectrum of diseases from mild to severe, and patients may be recruited from both inpatient or outpatient settings. Biases could persist under the test-negative design if broad variation in disease manifestation gives rise to differential healthcare-seeking behavior in terms of vaccination and care seeking for symptoms, and affect the probability of being tested.Citation20 Nevertheless, the choice of clinical symptoms for enrolled patients should ensure similarity between cases and controls and not be dependent on the vaccination or important confounders of the vaccination–disease relationship. Test-negative studies have typically, but not always, included persons seeking medical care during disease epidemic periods because calendar time is thus correlated with both vaccine uptake and with incidence of disease, analyses of test-negative data must control for calendar time.Citation14 Another source of bias is misclassification of disease status due to the imperfect sensitivity and specificity of test method3. And imperfect specificity of influenza testing caused greater bias in the test-negative design than did imperfect sensitivity.Citation21 In terms of distinguishing cases from controls, most TND studies defined vaccine-type positive as cases, where vaccines being evaluated were believed to have protective effect against the strains included in the vaccine. However, in the scenario where vaccination is thought to provide cross-protection against nonvaccine-types of the same pathogen, cases also included nonvaccine-types or all patients testing positive for the pathogen regardless of type. There is a lack of consensus on the choices of controls for each vaccine. A basic assumption of the test-negative design is that the risk of infections by nonvaccine-targeted causative pathogens resulting in similar clinical disease does not vary by vaccination status. This assumption may be violated if there is cross-protection in nonvaccine-type pathogens. Therefore, the choice to include vaccine-matched pathogens or not should depend on the public health question to be answered: are we interested in knowing by how much this vaccine will reduce the burden of the clinical disease of interest? Or are we interested in estimating the effectiveness of the vaccine only with respect to the pathogens it specifically targets? From a programmatic, policy, and public health perspective, the former argument is perhaps more relevant.Citation19In general case–control studies, controls must be representative of the source population. As such, in test-negative designs controls should belong to the same source population from which cases were identified, and they should be individuals who probably would have been identified as cases got the targeted pathogen of interest. Test-negative controls are chosen from patients seeking medical care with similar clinical illness and tested negative from laboratory diagnosis. They provide reassurance that they emerge from the same source population as cases, and would have been suspected cases with the targeted pathogen. The main advantage of the TND is its convenience of access to get controls representative of the source population. However, TND has its assumptions. The core assumption of the TND is that vaccination does not affect the probability of other pathogen infection. The controls with EV-71 negative but other enterovirus positive may affect the validity and accuracy of TND.Citation22 One of our limitations is that we did not demonstrate that EV-71 vaccine had no effect on EV-71-negative but other Enterovirus-positive disease with sensitivity analysis, although there had been studies indicating that EV-71 vaccine did not have cross-protection on other enterovirus.Citation6,Citation7 In influenza vaccine and rotavirus vaccine TND studies, most of them have sensitivity analysis with different control groups.Citation23,Citation24 Therefore, in EV-71 vaccine TND studies, sensitivity analysis is needed for further application.
Another limitation of our study is that we only have one RCT database. This RCT only contains participants aged from 6 to 35 months. In China, there are three kinds of EV-71 vaccine, two of which for 6–35 months and one of which for 6–71 months.Citation6–8 In the EV-71 vaccine effectiveness studies by TND,Citation5,Citation9,Citation10 between cases and controls, age level had significant statistical difference and was adjusted by multiple regression. They did not analyze the influence may caused by vaccine types although there has been a research shows that children aged 3–5 years old had non-inferior immunogenicity to that in infants aged 6–35 months.Citation25 In the RCT, characteristics between vaccine group and placebo group were balanced because of randomization. In this study, characteristics between cases and controls have no statistical differences and we didn’t adjust potential variates that may cause bias. Possibility it was from RCT database so cases and controls were balanced at the baseline. However, in general TND studies, cases, and controls usually have different baselines like gender composition, age level, or others.Citation26,Citation27 Thence, multivariate logistic regression is usually used to adjust covariates like sex, age, education level, calendar time, and so on.
In conclusion, we have validated the TND approach for EV-71 vaccine by using RCT database. TND, as an observational design, appears valid in EV-71 vaccine effectiveness assessment but requires further consideration in confounding and bias.
Authors’ contributions
Fengcai Zhu and Jingxin Li contributed to conception and design of the study; Li Zhang and Mingwei Wei contributed to collecting and analyzing the data; Li Zhang draft the manuscript; Jingxin Li, Mingwei Wei and Pengfei Jin revised the manuscript; All authors read and approved the final manuscript.
Disclosure of potential conflicts of interest
No potential conflicts of interest were disclosed.
Additional information
Funding
References
- Yang B, Liu F, Liao Q, Wu P, Chang Z, Huang J, Long L, Luo L, Li Y, Leung GM, et al. Epidemiology of hand, foot and mouth disease in China, 2008 to 2015 prior to the introduction of EV-A71 vaccine. Euro surveillance: bulletin Europeen sur les maladies transmissibles = European communicable disease bulletin 2017; 22(50):16–00824. PMID: 29258646, PMCID: PMC5743100. doi:10.2807/1560-7917.ES.2017.22.50.16-00824.
- Liu SL, Pan H, Liu P, Amer S, Chan TC, Zhan J, Huo X, Liu Y, Teng Z, Wang L, et al. Comparative epidemiology and virology of fatal and nonfatal cases of hand, foot and mouth disease in mainland China from 2008 to 2014. Rev Med Virol. 2015;25:115–28. Epub 2015 Feb 20, PMID: 25704797. doi:10.1002/rmv.1827.
- Xing W, Liao Q, Viboud C, Zhang J, Sun J, Wu JT, Chang Z, Liu F, Fang VJ, Zheng Y, et al. Hand, foot, and mouth disease in China, 2008–12: an epidemiological study. Lancet Infect Dis. 2014;14:308–18. Epub 2014 Jan 31, PMID: 24485991, PMCID: PMC4035015. doi:10.1016/S1473-3099(13)70342-6.
- Abzug MJ. The enteroviruses: problems in need of treatments. J Infect. 2014;68(Suppl 1):S108–14. doi:10.1016/j.jinf.2013.09.020.
- Li Y, Zhou Y, Cheng Y, Wu P, Zhou C, Cui P, Song C, Liang L, Wang F, Qiu Q, et al. Effectiveness of EV-A71 vaccination in prevention of paediatric hand, foot, and mouth disease associated with EV-A71 virus infection requiring hospitalisation in Henan, China, 2017–18: a test-negative case-control study. Lancet Child Adolesc Health. 2019;3:697–704. Epub 2019 Jul 30, PMID: 31375313. doi:10.1016/S2352-4642(19)30185-3.
- Zhu F, Xu W, Xia J, Liang Z, Liu Y, Zhang X, Tan X, Wang L, Mao Q, Wu J, et al. Efficacy, safety, and immunogenicity of an enterovirus 71 vaccine in China. N Engl J Med. 2014;370:818–28. PMID: 24571754. doi:10.1056/NEJMoa1304923.
- Li R, Liu L, Mo Z, Wang X, Xia J, Liang Z, Zhang Y, Li Y, Mao Q, Wang J, et al. An inactivated enterovirus 71 vaccine in healthy children. N Engl J Med. 2014;370:829–37. PMID: 24571755. doi:10.1056/NEJMoa1303224.
- Zhu F-C, Meng F-Y, Li J-X, Li X-L, Mao Q-Y, Tao H, Zhang YT, Yao X, Chu K, Chen QH, et al. Efficacy, safety, and immunology of an inactivated alum-adjuvant enterovirus 71 vaccine in children in China: a multicentre, randomised, double-blind, placebo-controlled, phase 3 trial. The Lancet. 2013;381:2024–32. Epub 2013 May 29, PMID: 23726161. doi:10.1016/S0140-6736(13)61049-1.
- Wang X, An Z, Huo D, Jia L, Li J, Yang Y, Liang Z, Wang Q, Wang H. Enterovirus A71 vaccine effectiveness in preventing enterovirus A71 infection among medically-attended hand, foot, and mouth disease cases, Beijing, China. Hum Vaccin Immunother. 2019;15:1183–90.doi:10.1080/21645515.2019.1581539.
- Jiang L, Wang J, Zhang C, He W, Mo J, Zeng J, Chen M, Tan Y, Ning C. Effectiveness of enterovirus A71 vaccine in severe hand, foot, and mouth disease cases in Guangxi, China. Vaccine 2019;38(7):1804–09. Epub 2019 Dec 28, PMID: 31892446. doi: 10.1016/j.vaccine.2019.12.025.
- Zhang D. A template for EV-A71 vaccine evaluation in the real world. Lancet Child Adolesc Health. 2019;3:665–66. doi:10.1016/S2352-4642(19)30181-6.
- Foppa IM, Haber M, Ferdinands JM, Shay DK. The case test-negative design for studies of the effectiveness of influenza vaccine. Vaccine. 2013;31:3104–09. doi:10.1016/j.vaccine.2013.04.026.
- Araki K, Hara M, Tsugawa T, Shimanoe C, Nishida Y, Matsuo M, Tanaka K. Effectiveness of monovalent and pentavalent rotavirus vaccines in Japanese children. Vaccine. 2018;36:5187–93. Epub 2018 Jul 20, PMID: 30037664. doi:10.1016/j.vaccine.2018.07.007.
- Jackson ML, Nelson JC. The test-negative design for estimating influenza vaccine effectiveness. Vaccine. 2013;31:2165–68. doi:10.1016/j.vaccine.2013.02.053.
- De Serres G, Skowronski DM, Wu XW, Ambrose CS The test-negative design: validity, accuracy and precision of vaccine efficacy estimates compared to the gold standard of randomised placebo-controlled clinical trials. Euro surveillance: bulletin Europeen sur les maladies transmissibles = European communicable disease bulletin 2013; 18.
- WPRO | A Guide to Clinical Management and Public Health Response for Hand, Foot and Mouth disease (HFMD).
- Schwartz LM, Halloran ME, Rowhani-Rahbar A, Neuzil KM, Victor JC. Rotavirus vaccine effectiveness in low-income settings: an evaluation of the test-negative design. Vaccine. 2017;35:184–90. doi:10.1016/j.vaccine.2016.10.077.
- Sullivan SG, Feng S, Cowling BJ. Potential of the test-negative design for measuring influenza vaccine effectiveness: a systematic review. Expert Rev Vaccines. 2014;13:1571–91. doi:10.1586/14760584.2014.966695.
- Chua H, Feng S, Lewnard JA, Sullivan SG, Blyth CC, Lipsitch M, Cowling BJ. The Use of Test-negative Controls to Monitor Vaccine Effectiveness: A Systematic Review of Methodology. Epidemiology (Cambridge, Mass). 2020;31:43–64. PMID: 31609860, PMCID: PMC6888869. doi:10.1097/EDE.0000000000001116.
- Lewnard JA, Tedijanto C, Cowling BJ, Lipsitch M. Measurement of Vaccine Direct Effects Under the Test-Negative Design. Am J Epidemiol. 2018;187:2686–97. doi:10.1093/aje/kwy163.
- Jackson ML, Rothman KJ. Effects of imperfect test sensitivity and specificity on observational studies of influenza vaccine effectiveness. Vaccine. 2015;33:1313–16. doi:10.1016/j.vaccine.2015.01.069.
- Fukushima W, Hirota Y. Basic principles of test-negative design in evaluating influenza vaccine effectiveness. Vaccine. 2017;35:4796–800. doi:10.1016/j.vaccine.2017.07.003.
- Bar-Zeev N, Kapanda L, Tate JE, Jere KC, Iturriza-Gomara M, Nakagomi O, Mwansambo C, Costello A, Parashar UD, Heyderman RS, et al. Effectiveness of a monovalent rotavirus vaccine in infants in Malawi after programmatic roll-out: an observational and case-control study. Lancet Infect Dis. 2015;15:422–28. PMID: 25638521, PMCID: PMC4374102. doi:10.1016/S1473-3099(14)71060-6.
- Rolfes MA, Flannery B, Chung J, O’Halloran A, Garg S, Belongia EA, Gaglani M, Zimmerman RK, Jackson ML, Monto AS, et al. Effects of Influenza Vaccination in the United States during the 2017-2018 Influenza Season. Clin Infect Dis 2019;69(11):1845–53. PMID: 30715278; PMCID: PMC7188082. doi: 10.1093/cid/ciz075.
- Gu W, Zeng G, Hu Y-M, Hu Y-S, Zhang Y, Hu Y-L, Wang Y, Li JX, Zhu FC. A comparative analysis of immunogenicity and safety of an enterovirus 71 vaccine between children aged 3-5 years and infants aged 6-35 months. Expert Rev Vaccines. 2018;17:257–62. Epub 2018 Jan 29, PMID: 29363365. doi:10.1080/14760584.2018.1430572.
- Flannery B, Chung JR, Monto AS, Martin ET, Belongia EA, McLean HQ, Gaglani M, Murthy K, Zimmerman RK, Nowalk MP, et al. Influenza Vaccine Effectiveness in the United States during the 2016-2017 Season. Clinical infectious diseases: an official publication of the Infectious Diseases Society of America 2018. Clin Infect Dis. 2019;68(11):1798–1806. PMID: 30204854, PMCID: PMC6522684. doi: 10.1093/cid/ciy775.
- Jackson ML, Chung JR, Jackson LA, Phillips CH, Benoit J, Monto AS, Martin ET, Belongia EA, McLean HQ, Gaglani M, et al. Influenza Vaccine Effectiveness in the United States during the 2015-2016 Season. N Engl J Med. 2017;377:534–43. PMID: 28792867, PMCID: PMC5727917. doi:10.1056/NEJMoa1700153.