https://orcid.org/0000-0002-1644-0006
ABSTRACT
Since TND could be an appropriate method to assess vaccine effectiveness, we want to know whether it may be used for the estimation of vaccine immunological surrogate endpoints, like case-control study. We conducted two study designs (test-negative design (TND) VS matched case-control design (MCC)) to evaluate immunological surrogate endpoint against EV71-associated diseases. We calculated sensitivity (proportion of participants with EV71-associated disease who have a titer less than the cutoff at day 56), specificity (proportion of matched controls who have a titer equal or greater than the cutoff at day 56), and corresponding Youden index ([sensitivity + specificity] − 1). Then, we compared them between TND and MCC. In test-negative design, we totally enrolled 7029 subjects, 49 tested positive as cases and 6980 tested negative as controls in per-protocol population. In matched case-control design, we totally enrolled 305 subjects, 51 as cases, and 254 as controls in whole cohort. In sensitivity and specificity comparison, TND and MCC’s results were similar to each other, except for a titer of 1:4. Nonetheless, in Youden index comparison, MCC’s results were slightly higher than the TND’s, except for a titer of 1:4. EV71 vaccine immunological surrogate endpoints derived from TND was similar to MCC’s. Our results supported that TND could become an alternative research design with the progress of surveillance.
Introduction
Enterovirus 71 (EV71) is one of the major pathogens causing hand, foot, and mouth disease (HFMD), causing a substantial disease burden in the Asia-Pacific region particularly in young children under 5 years old.Citation1 Currently, there is no specific treatment for HFMD caused by EV71.Citation2 Three inactivated monovalent EV71 vaccines (developed by Chinese Academy of Medical Sciences(CAMS), Sinovac Biotech, and Beijing Vigoo Biological) were licensed in China since 2016, which showed high efficacy (94.8–97.4%) against EV71-associated HFMD but no cross-protection against HFMD caused by other serotypes in children.Citation3–5 Therefore, other kinds of enterovirus vaccine, such as monovalent Coxsackievirus 16(CA-16) vaccine and bivalent EV71 and CA-16 vaccine, are under development.Citation6,Citation7
An applicable immunological surrogate of protection would help to avoid new large trials and facilitate getting new products and formulations approved. It also could reduce the sample size or shorten the duration of a clinical trial. In previous studies,Citation3,Citation5 correlate of immunity against EV71-associated disease was explored. The results from matched case-control study design and a receiver operating characteristics (ROC) curve analysis suggested that a potential titer of neutralizing antibody (NTAb) of 1:16-1:32 at day 56 was proposed to be an immunological surrogate endpoint for the EV71 vaccine protection.
Test-negative design (TND) is considered as a variant of the traditional case–control study and has become popular for post-licensure observational studies of the effectiveness of vaccines, especially for influenza vaccines.Citation8–10 Several studies have operated a comparison between the test-negative design and traditional case-control design for estimation of vaccine effectiveness.Citation11–13 These studies prompt that TND, compared with traditional case–control study, is a convenient and reliable alternative for estimation of vaccine effectiveness.
Since TND could be an appropriate method to assess vaccine effectiveness, we want to know whether it may be used for estimation of vaccine immunological surrogate endpoints, like case-control study. Hence, we practice a comparison of immunological surrogate endpoints derived from test-negative design and matched case-control study design.
Method
Data source
The database is from a multi-center, randomized, double-blind, placebo-controlled, phase 3 trial which conducted in China (NCT01508247). This first EV71 vaccine phase 3 trial performed by Vigoo had high quality process and data. The trial enrolled 10,245 children and infants, 5120 participants in vaccine group, and 5125 in placebo group. Details can be found in the previous publication.Citation5
Matched case-control study design
To evaluate the correlation between EV71 neutralizing antibody levels and disease protection, we assessed correlates of immunity for all participants with confirmed EV71 and their matched case-free participants who had data for antibody titer at day 56. EV71 vaccine immune procedure is at day 0 and 28 with intramuscular injection. There is a popular belief that antibody levels approach the peak level at 28 days (or a month) after a complete immune procedure. Therefore, we chose Day 56 as the cutoff day. The diseases caused by EV71 infection were diagnosed as the case group, and each case was matched at 1:5 after the initial diagnosis to form a subgroup. The matching conditions were as follows: (1) during the period from 0 days to 14 months after the first dose of vaccination, the subjects who had no diseases caused by EV71, and if a control group found diseases caused by EV71 infection during the subsequent surveillance period, the control group was converted to a case and was excluded from the control group; (2) the age gap between the case and the case was less than 3 months and lived in the same village or neighboring village.
Test-negative design
In test-negative design, patients seeking medical care for a predefined clinical condition are tested for a specific viral infection by using a highly sensitive and specific laboratory test. Those tested positive are cases and controls are tested negative for infection but meeting the same enrollment criteria. In origin phase III trial, we did a high-quality combination of active and passive monitoring to capture any suspected EV71 cases. Participants were actively followed after being administered two doses of vaccine or placebo. Guardians were encouraged to take their children to treatment in clinics or hospitals for any illness. The surveillance was from 56 days after first dose to 14 months.
All potential cases were reported by the clinics or hospitals. Throat or rectal swabs or both were taken for pathogen detection within 24 h. These samples were tested for EV71 by real-time PCR (fluorescence assay) with a viral RNA diagnostic kit. A participant was defined as a case if at least one positive test result during follow-up. A participant was defined as a control if at least one negative test results during follow-up and the participant had no positive results.
Measurement of EV71 neutralizing antibody
Serum samples of 3 mL were collected from subjects on day 56 (day 28 after second dose) for determination of EV71 NTAb titers. Serums samples were stored at a temperature of −20∘C or below before tested. A modified cytopathogenic effect method (CPE method) was used to measure EV71 NTAb titers.Citation14 In brief, samples were 2-fold serially diluted from 1:8 to 1:16384 and NTAb titers were defined as the highest dilution capable of inhibiting 50% of the CPEs. Antibody titers lower than 1:8 was assigned values of 1:4 for calculation. 1219 serum samples from immunogenicity cohort collected on day 56 had been detected by National Institutes for Food and Drug Control (NIFDC), and remaining 8435 serum samples were measured by Beijing Vigoo Biological Ltd later using the same method. The tested original NTAb titers from NIFDC and Beijing Vigoo were standardized according to the NTAb titers (U/mL) of the reference serum (N12: 1000U). If Nab measure were performed with different personnel or multiple times, the results of final measurements would be different and affected our results. Thus, to avoid final measurements affected our results, a standardization method would convert the tested NTAb titers from different laboratories into standardized NTAb titers. This method was adapted from the calibration methods applied for polio previously.Citation15
Statistical analysis
In test-negative design, we evaluated immunological surrogate endpoints based on the per-protocol population in common with previous work about immune correlates,Citation16,Citation17 which consisted of all eligible participants who completed the 2-dose regimen and with available EV71 NTAb titers in serum post-vaccination. Matched case-control study (MCC) analysis has been performed in origin phase III trial.Citation5
We calculated sensitivity (proportion of participants with EV71-associated disease who have a titer less than the cutoff at day 56), specificity (proportion of matched controls who have a titer equal or greater than the cutoff at day 56), and corresponding Youden index ([sensitivity + specificity] − 1). The cutoff with the maximum Youden index could be thought of as a surrogate of protection for it could provide the clearest distinction between cases and controls. We used the χ2 test or Fisher’s exact test to analyze categorical data, ANOVA to analyze continuous data. Statistical analyses were done with SAS software (version 9.4). All statistical tests were two-sided, and statistical significance was defined as P < 0.05.
Results
We conducted two study designs to evaluate immunological surrogate endpoint against EV71-associated diseases (). In test-negative design, we totally enrolled 7029 subjects, 49 tested positive as cases and 6980 tested negative as controls in per-protocol population. In matched case-control design, we totally enrolled 305 subjects, 51 as cases and 254 as controls in whole cohort. A part of baseline data had been reported in previous research.Citation18
Figure 1. Study plan.
showed the characteristics of cases and controls in test-negative design and matched case-control design. There are no significant statistical differences between cases and controls on sex, age, height and weight, both in test-negative design and matched case-control design. However, the case’s GMT and the ratio of received vaccine numbers were significantly lower than the control’s, also both in test-negative design and matched case-control design.
Table 1. Characteristics of cases and controls in test-negative design and matched case-control design.
The percentage of controls and cases about TND and MCC with titer ≥1:2 was shown in . Clearly, the levels of anti-EV71 neutralizing antibody titers in control group were higher than case groups. shows the sensitivity and specificity of different cutoff neutralizing antibody titer against EV71associated disease by test-negative design and matched case-control design. Except 1:4 neutralizing antibody titer, the TND results of different cutoff were close to the MCC’s results. The maximum Youden index was provided by a titer of 1:32 in TND while the maximum Youden index was reached at a titer of 1:16 in MCC. However, both in TND and MCC, accounting for sensitivity, specificity, and Youden index, a titer of between 1:16 and 1:32 seemed to provide the best protection against EV71-associated diseases.
Figure 2. Reverse cumulative curves for enterovirus 71 (EV71) neutralizing antibody titers about TND and MCC.
Table 2. Sensitivity and specificity of different cutoff neutralizing antibody titer against EV71associated disease by test-negative design and matched case-control design.
revealed the comparison of sensitivity, specificit,y and Youden index between the TND and MCC. In sensitivity and specificity comparison, TND and MCC’s results were similar to each other. Nonetheless, in Youden index comparison, MCC’s results were slightly higher than the TND’s, except for a titer of 1:4.
Figure 3. The comparison of sensitivity, specificity, and Youden index between the TND and MCC.
Discussion
We tried to demonstrate the feasibility of TND for estimation of vaccine immunological surrogate endpoints. Taking EV71 vaccine as an example, we did a receiver operating characteristics (ROC) curve analysis in both TND and MCC. We compared the Youden index between TND and MCC and found that TND’s result was similar to MCC’s.
An immune correlate of protection is an immunological response (either humoral or cellular) that reliably predicts the level of vaccine efficacy to evaluate a clinically meaningful endpoint.Citation19 There have been some studies to evaluate EV71 vaccine immunological surrogate endpoints. Jin and his colleagues fitted scaled logit model to assess the mechanistic correlate of protection.Citation17 They found that NTAb with a titers of 14.7 U/ml (around 1:16) post-vaccination can be validated as a surrogate associated with the protection of 50% against EV71-associated disease. Besides, Zhu and his colleagues used logistic regression to analyze some other factors as covariates, including the NTAb titer level at baseline, age, and sex, which may affect the efficacy of the EV71 vaccine.Citation16 They got a slightly higher NTAb titer of 26.6 U/ml (around 1:30) as the immunological surrogate endpoint for EV71 vaccine protection by excluding subjects with a positive baseline in the model. Compared with TND or case-control study, mathematical model could be more precise for estimation of vaccine immunological surrogate endpoints since data in the subset of cohort using ROC curve analysis was limited and potentially biased by the selection of the controls. But we consider that TND and case-control study still could be used as tentative exploration designs.
Many previous studies have discussed the advantages and disadvantages about the TND and traditional case-control study for estimating vaccine effectiveness. Case-control study can be resource-efficient and particularly useful for relatively uncommon diseases because efforts are focused on accurately ascertaining disease status and vaccination history for a relatively small number of cases and controls. Despite being widely used to evaluate vaccine performance, the case-control methodology is susceptible to bias and confounding largely due to controls selection.Citation20,Citation21 In like manner, TND is often suspicious of validity and accuracy although it is popular because of ease of implementation. However, most vaccine effectiveness studies showed that TND was almost the same as the traditional case-control study, even TND produced less biased vaccine effectiveness estimates.Citation22,Citation23 Our study results still gained a similar conclusion that EV71 vaccine immunological surrogate endpoint derived from TND was the same as case-control study’s. But TND’s Youden index seems lower than the case-control study’s. Maybe it was because that we did not use the same number of cases or the controls quantity varied greatly.
As far as we know, this is the first analysis for the comparison of the test-negative and the matched case-control study designs for estimation of vaccine immunological surrogate endpoints. But we just did a simple analysis using EV71 vaccine as example to demonstrate the validity of TND for vaccine immunological surrogate endpoints. More study and development of this idea are left to future work. Before our study, Dean A. Follmann. Et detailed an approach to immune correlates analysis under a test negative design using samples collected from subjects received Ebola vaccine (Vesicular Stomatitis Vaccine, VSV).Citation24 They contributed the idea that an estimate of the effect of the relevant immune response on the risk of disease can be estimated by using an irrelevant immune response based on a prediction model that is estimated in vaccinated controls using samples collected upon arrival at a clinic. In vaccine effectiveness TND study, a key assumption is that the incidence of interest disease is the same in the vaccinated and unvaccinated groups within care-seeking behavior strata and vaccination will not affect non-vaccine pathogen.Citation25 So similarly, in immune correlates TND study, the key requirement is that the vaccine of interest produce irrelevant immune responses that will be unaffected by the disease of interest, and that these immune responses are correlated with the immune response of interest.Citation24
There were some limitations to this study. First, in TND, we did not enroll those subjects who receive only one dose because the previous study which fitted scaled logit model to assess the mechanistic correlate of protection used per-protocol population. But in MCC, we used whole cohorts. Therefore, the number of cases in TND was smaller than that in MCC. It may have a slight effect on the results’ comparison. Second, TND and MCC were different in selecting controls. It may affect the baseline comparability of the two study designs for the negative controls were much more than matched controls. However, it didn’t affect the final result that the optimal Youden index was 1:16-1:32. Third, we excluded those subjects who had no NAb titer data at Day 56. This could cause potential selection bias. Fortunately, missing data accounted for only a small fraction. Finally, this study was a post-hoc analysis and we did not make a sample size estimation.
Compared with vaccine effectiveness TND study which requires vaccination and disease data, immune correlates TND study also requires long-term, uniform, standardized storage of samples from hundreds of thousands of subjects just after vaccination. As a result, the biggest challenge in this approach is whether the antibody data is amenable. And although it is difficult to use TND to evaluate immunological surrogate endpoints after the vaccine is marketed, our results supported that TND could still become an alternative research design, especially with the development of the vaccination system, case surveillance network, and more routine antibody monitoring in the future.
Authors’ contributions
Fengcai Zhu and Jingxin Li contributed to conception and design of the study; Li Zhang and Pengfei Jin Wei contributed to collecting and analyzing the data; Li Zhang drafted the manuscript; Jingxin Li, Mingwei Wei， Pengfei Jin and Hudachuan Jiang revised the manuscript; all authors read and approved the final manuscript.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Additional information
Funding
References
- Yang B, Liu F, Liao Q, Wu P, Chang Z, Huang J, Long L, Luo L, Li Y, Leung GM, et al. Epidemiology of hand, foot and mouth disease in China, 2008 to 2015 prior to the introduction of EV-A71 vaccine. Eurosurveillance. 2017;22(50): 16–6. doi:10.2807/1560-7917.ES.2017.22.50.16-00824.
- Abzug MJ. The enteroviruses: problems in need of treatments. J Infect. 2014;68(Suppl 1):S108–14. doi:10.1016/j.jinf.2013.09.020.
- Zhu F, Xu W, Xia J, Liang Z, Liu Y, Zhang X, Tan X, Wang L, Mao Q, Wu J, et al. Efficacy, safety, and immunogenicity of an enterovirus 71 vaccine in China. N Engl J Med. 2014;370(9):818–28. doi:10.1056/NEJMoa1304923.
- Li R, Liu L, Mo Z, Wang X, Xia J, Liang Z, Zhang Y, Li Y, Mao Q, Wang J, et al. An inactivated enterovirus 71 vaccine in healthy children. N Engl J Med. 2014;370(9):829–37. doi:10.1056/NEJMoa1303224.
- Zhu F-C, Meng F-Y, Li J-X, Li X-L, Mao Q-Y, Tao H, Zhang Y-T, Yao X, Chu K, Chen Q-H, et al. Efficacy, safety, and immunology of an inactivated alum-adjuvant enterovirus 71 vaccine in children in China: a multicentre, randomised, double-blind, placebo-controlled, phase 3 trial. Lancet. 2013;381(9882):2024–32. doi:10.1016/s0140-6736(13)61049-1.
- Fan S, Liao Y, Jiang G, Wang L, Zhao H, Yu L, Xu X, Li D, Zhang Y, Li Q, et al. Efficacy of an inactivated bivalent vaccine for enterovirus 71 and coxsackievirus A16 in mice immunized intradermally. Vaccine. 2021;39(3):596–604. doi:10.1016/j.vaccine.2020.11.070.
- Chen K, Li C, Wang Y, Shen Z, Guo Y, Li X, Zhang Y. Optimization of Vero cells grown on a polymer fiber carrier in a disposable bioreactor for inactivated Coxsackievirus A16 vaccine development. Vaccines. 2021;9(6):613. doi:10.3390/vaccines9060613.
- Jackson ML, Nelson JC. The test-negative design for estimating influenza vaccine effectiveness. Vaccine. 2013;31(17):2165–68. doi:10.1016/j.vaccine.2013.02.053.
- Foppa IM, Haber M, Ferdinands JM, Shay DK. The case test-negative design for studies of the effectiveness of influenza vaccine. Vaccine. 2013;31(30):3104–09. doi:10.1016/j.vaccine.2013.04.026.
- Fukushima W, Hirota Y. Basic principles of test-negative design in evaluating influenza vaccine effectiveness. Vaccine. 2017;35(36):4796–800. doi:10.1016/j.vaccine.2017.07.003.
- Haber M, Lopman BA, Tate JE, Shi M, Parashar UD. A comparison of the test-negative and traditional case-control study designs with respect to the bias of estimates of rotavirus vaccine effectiveness. Vaccine. 2018;36(33):5071–76. doi:10.1016/j.vaccine.2018.06.072.
- Araki K, Hara M, Shimanoe C, Nishida Y, Matsuo M, Tanaka K. Case-control study of Rotavirus vaccine effectiveness compared to test-negative controls or hospital controls. J Epidemiol. 2018;29(8):282–87. doi:10.2188/jea.JE20180054.
- Shi M, An Q, Ainslie KEC, Haber M, Orenstein WA. A comparison of the test-negative and the traditional case-control study designs for estimation of influenza vaccine effectiveness under nonrandom vaccination. BMC Infect Dis. 2017;17(1):757. doi:10.1186/s12879-017-2838-2.
- Liang Z, Mao Q, Gao Q, Li X, Dong C, Yu X, Yao X, Li F, Yin W, Li Q, et al. Establishing China’s national standards of antigen content and neutralizing antibody responses for evaluation of enterovirus 71 (EV71) vaccines. Vaccine. 2011;29(52):9668–74. doi:10.1016/j.vaccine.2011.10.018.
- Chen YJ, Meng FY, Mao Q, Li J-X, Wang H, Liang Z-L, Zhang Y-T, Gao F, Chen Q-H, Hu Y, et al. Clinical evaluation for batch consistency of an inactivated enterovirus 71 vaccine in a large-scale phase 3 clinical trial. Hum Vaccines Immunother. 2014;10(5):1366–72. doi:10.4161/hv.28397.
- Zhu W, Jin P, Li JX, Zhu FC, Liu P. Correlates of protection for inactivated enterovirus 71 vaccine: the analysis of immunological surrogate endpoints. Expert Rev Vaccines. 2017;16(9):945–49. doi:10.1080/14760584.2017.1335603.
- Jin P, Li J, Zhang X, Meng F, Zhou Y, Yao X, Gan Z, Zhu F. Validation and evaluation of serological correlates of protection for inactivated enterovirus 71 vaccine in children aged 6-35 months. Hum Vaccines Immunother. 2016;12(4):916–21. doi:10.1080/21645515.2015.1118595.
- Zhang L, Wei M, Jin P, Li J, Zhu F. An evaluation of a test-negative design for EV-71 vaccine from a randomized controlled trial. Hum Vaccines Immunother. 2021;17(7):2101–06. doi:10.1080/21645515.2020.1859900.
- Sadoff JC, Wittes J. Correlates, surrogates, and vaccines. J Infect Dis. 2007;196(9):1279–81. doi:10.1086/522432.
- Verani JR, Baqui AH, Broome CV, Cherian T, Cohen C, Farrar JL, Feikin DR, Groome MJ, Hajjeh RA, Johnson HL, et al. Case-Control vaccine effectiveness studies: preparation, design, and enrollment of cases and controls. Vaccine. 2017;35(25):3295–302. doi:10.1016/j.vaccine.2017.04.037.
- Verani JR, Baqui AH, Broome CV, Cherian T, Cohen C, Farrar JL, Feikin DR, Groome MJ, Hajjeh RA, Johnson HL, et al. Case-Control vaccine effectiveness studies: data collection, analysis and reporting results. Vaccine. 2017;35(25):3303–08. doi:10.1016/j.vaccine.2017.04.035.
- Balasubramani GK, Zimmerman RK, Eng H, Lyons J, Clarke L, Nowalk MP. Comparison of local influenza vaccine effectiveness using two methods. Vaccine. 2021;39(8):1283–89. doi:10.1016/j.vaccine.2021.01.013.
- Franke MF, Jerome JG, Matias WR, Ternier R, Hilaire IJ, Harris JB, Ivers LC. Comparison of two control groups for estimation of oral cholera vaccine effectiveness using a case-control study design. Vaccine. 2017;35(43):5819–27. doi:10.1016/j.vaccine.2017.09.025.
- Follmann DA, Dodd L. Immune correlates analysis using vaccines from test negative designs. Biostatistics. 2020. doi:10.1093/biostatistics/kxaa037.
- Sullivan SG, Tchetgen Tchetgen EJ, Cowling BJ. Theoretical basis of the test-negative study design for assessment of influenza vaccine effectiveness. Am J Epidemiol. 2016;184(5):345–53. doi:10.1093/aje/kww064.