https://orcid.org/0000-0002-3128-649X
ABSTRACT
Introduction: Enterovirus A71(EV-A71)-associated hand, foot, and mouth disease (HFMD) has been reported worldwide, and poses a particularly heavy burden on patients, families, and society in China. Three Chinese companies have licensed inactivated EV-A71 vaccines, all of which have demonstrated good efficacy for preventing EV-A71-associated disease in clinical trials. However, real-world performance of EV-A71 vaccine has not been evaluated.
Methods: We used a test-negative design case-control study to estimate vaccine effectiveness (VE) against medically attended EV-A71-associated HFMD. Subjects were children 5 years of age and under who had been in health facilities participating in the HFMD case and virologic surveillance platforms in Beijing. Enterovirus infections were laboratory confirmed, and EV-A71 vaccination status was extracted from electronic immunization records. Children testing positive for EV-A71 were cases; controls were children testing negative for EV-A71 infection. Logistic regression was used to estimate VE. We assessed sensitivity of VE estimates to control group inclusion criteria by repeating the regression analyses with two alternative control groups.
Results: A total of 2,184 HFMD patients aged 5 years and under were enrolled in the study; 24 were severe, and 2,160 were mild. For severe cases, two-dose VE estimate was 100% (95% CI: −68.1%, 100%). For mild cases, 1-dose and 2-dose adjusted VE estimates were 69.8% and 83.7%, respectively. Two-dose VE estimates varied by less than 4 percentage points regardless of control group definition.
Conclusions: Our findings suggested the vaccines performed well in the real world for children 5 years of age and under in Beijing, China.
Introduction
Hand, foot, and mouth disease (HFMD) is a viral illness characterized by fever, painful sores in the mouth, and rash or blisters on the hands, feet, and buttocks. It is a common childhood illness that is caused by serotypes of the A species of the Enterovirus genus in the Picornaviridae family.Citation1 Severe and fatal cases have mainly been caused by enterovirus A71 (EV-A71) infection of children, especially children less than 3 years of age. Outbreaks of HFMD caused by EV-A71 (EV-A71-HFMD) have been reported worldwide, but are prominent in the Asia-Pacific region.Citation2-Citation5 China reports the majority of HFMD cases globally, and since 2010, HFMD has been the top-ranked notifiable disease in China.Citation6 Between 2008 and 2012, EV-A71 accounted for more than 90% of laboratory-confirmed, fatal HFMD cases reported in China.Citation7 EV-A71-HFMD morbidity and mortality causes great economic and psychological burdens on patients, families, and society. One severe HFMD case can incur over $3,000 USD in health care costs; every 1,000 cases will lead to a loss of 13 quality-adjusted life-years (QALY).Citation8 Because of the young age of infection, fatal EV-A71-HFMD cases impose high economic losses due to premature death, ranging from $80,000 to $150,000 USD per fatal case.Citation9
Three Chinese vaccine manufacturers (Chinese Academy of Medical Sciences (CAMS), Sinovac Biotech, and Beijing Vigoo Biological) developed inactivated EV-A71 vaccines and were licensed by the China Medical Products Authority to market their vaccines in China. The vaccines are given in two doses, administered 4 weeks apart to children aged 6 to 59 months. Since there is no treatment for EV-A71-HFMD, the debut of these vaccines ushers in a new era of prevention.
Beijing municipality began vaccinating children with EV-A71 vaccine in August 2016, and by the end of December 2017, almost 140,000 (15%) children between 6 and 59 months had received two doses of the vaccine. Although large clinical trials with more than 10,000 children showed at least 90% efficacy for preventing EV-A71-HFMD,Citation10-Citation12 real-world performance of EV-A71 vaccines has not been evaluated. Currently, EV-A71 vaccines are not included in the Chinese government’s Expanded Program on Immunization. Because governmental public health immunization programs and policies must be based upon post-licensure effectiveness, understanding the real-world performance of EV-A71 vaccines is critically important.
Test-negative design (TND) case-control studies have been used to estimate real-world vaccine effectiveness (VE) of influenza and rotavirus vaccines using sentinel surveillance platforms.Citation13-Citation15 Case-based and virological HFMD surveillance platforms have been operating in Beijing since 2007,Citation16 raising the possibility of using a TND study to estimate effectiveness of EV-A71vaccine. We report results of a TND VE study of EV-A71 vaccine conducted in Beijing in 2017.
Results
Subjects
Between January and December 2017, we enrolled 2,697 medically-attended HFMD cases; 513 cases met exclusion criteria and were excluded: 374 were born outside the predefined date-of-birth range, 10 were less than 6 months old when clinical diagnosed with HFMD, 11 had tested positive for EV-A71 before being vaccinated, 25 received a second dose of EV-A71 vaccine less than 28 days before illness onset, 44 resided in Beijing less than 6 months, and 49 were not registered in the Beijing Management System of Information for the Immunization Program (BMSIIP). After exclusions, 2,184 HFMD cases remained, of which 24 (1.1%) were severe cases and 2,160 (98.9%) were mild cases (). Among severe cases, 18 (75.0%) tested positive for enterovirus, and among these, 7 (38.9%) were positive for EV-A71. Among mild cases, 1,528 (70.7%) tested positive for an enterovirus, and 368 (24.1%) of these were positive for EV-A71.
Figure 1. Flow chart of subject enrollment in the test-negative design case-control study for the estimates of EV-A71 vaccine effectiveness during 2017, in Beijing, China.
Note: HFMD: hand, foot, and mouth disease. EV-A71: enterovirus A71. BMSIIP: Beijing Management System of Information for the Immunization Program. VE: vaccine effectiveness.
shows the monthly distribution of HFMD cases by test status and serotype. shows the number of two-dose EV-A71 vaccination completions by month. Most EV-A71 vaccination occurred April to June. shows a comparison between vaccinated and unvaccinated mild cases. There was no significant difference in vaccination status by sex. Younger cases (aged 6–35 months) were more likely to have been vaccinated (p < 0.001). Samples were usually obtained within 3 days of illness onset, and there was no significant difference in vaccination status by the intervals between illness onset and obtaining samples. Vaccinated cases were significantly less likely to be infected with EV-A71 than those not vaccinated (4.5% vs. 18.0%, p < 0.001). Most specimens were throat swab (83.6%), and there was no significant difference in vaccination status among different types of specimens (p = 0.151). Between the vaccinated and non-vaccinated group, no statistically significant difference was observed in the infection of other EVs (CV-A16, CV-A6, or other non-EV-A71, non-CV-A16 and non-CV-A6 enteroviruses).
Table 1. Comparison of demographic characteristics and virus infection between mild vaccinated and non-vaccinated HFMD cases.
Figure 2. Monthly number of the cases testing negative and positive for enterovirus by serotype.
Note: EV-A71: enterovirus A71. CV-A16: coxsackievirus A16. CV-A6: coxsackievirus A6. Others: otherenterovirus than EV-A71, CV-A16 and CV-A6. Negative: negative for all enteroviruses.
Figure 3. Timeline of number of subjects receiving two doses of EV-A71 vaccine.
Note: EV-A71: enterovirus A71. The number of subjects who received EV-A71 vaccine were excluded: (1) who only received one dose of EV-A71 vaccine, and (2) who received a second dose of EV-A71 vaccine that was <28 days before the illness onset.
shows that EV-A71-positive cases and EV-A71-negative controls did not differ by sex (p = 0.325). The proportion of subjects aged 6–35 months was lower among the EV-A71-positive cases than the EV-A71-negative controls (26.1% vs. 32.5%, p = 0.016). No statistically significant difference between cases and controls was observed in the intervals between illness onset to sampling and different types of specimens.
Table 2. Comparison of demographic characteristics and virus infection between mild EV-A71-positive cases and EV-A71-negative controls.
EV-A71 vaccine effectiveness
Among severe HFMD cases, none of EV-A71-positive cases had received EV-A71 vaccine, while 4 (23.5%) of 17 EV-A71 negative HFMD controls had received two doses. The overall VE against EV-A71 associated severe HFMD cases was estimated at 100% (95% confidence interval [CI]: −68.1%, 100%).
shows VE estimates against mild, medically-attended EV-A71-HFMD cases using EV-A71-negative controls. Included in the control group are all subjects who tested negative for EV-A71 infection, regardless of positive or negative for any other enteroviruses. The overall crude VE for one dose was 70.3% (95% CI: 4.0%, 90.8%), and VE adjusted for sex, age group (6–35 months, 36–59 months), and calendar month, was 69.8% (95% CI: 1.6%, 90.7%). The overall crude VE for two doses was 82.3% (95% CI: 51.7%, 93.5%) and the adjusted VE was 83.7% (95% CI: 54.9%, 94.1%). The adjusted one-dose VE was 68.2% (95% CI: −143.0%, 95.8%) for the 6–35-month group and 69.6% (95% CI: −29.9%, 92.9%) for the 36–59 month group. And the adjusted two-dose VE was 77.3% (95% CI: 4.6%, 94.6%) for the 6–35-month group and 86.8% (95% CI: 44.6%, 96.9%) for the 36–59 month group.
Table 3. Crude and adjusted estimates of vaccine effectiveness against mild, medically-attended EV-A71-HFMD for different number of doses and age group.
We found that estimated VEs did not change significantly with different control group criteria. Differences in adjusted VEs varied by less than 4 percentage points from the primary analysis control group for each of the alternative control group definitions [–].
Table 4. Estimates of vaccine effectiveness in sensitivity analysis, using controls who were positive for an enterovirus other than EV-A71.
Table 5. Estimates of vaccine effectiveness in sensitivity analysis, using pan-EV negative controls.
Match between vaccine virus and circulating virus
Fourteen EV-A71 strains that circulated in Beijing in 2016 and 2017 were isolated and sequenced; all belonged to C4a, which clustered with the vaccine strains (FY23/AH/CHN/2008, FY08-C30-P2/AH/CHN/2008, FY7VP5/AH/CHN/2008).See .
Figure 4. Phylogentic analysis of VP1 gene of EV-A71 strains from HFMD virological surveillance during the 2016–2017 in Beijing, China#.
Note: EV-A71: enterovirus A71.HFMD: hand, foot, and mouth disease.# The EV-A71 strains analyzed in this study were indicated with solid triangles and squares, and the vaccine strains were shown with solid dots. ▲the strains isolated in 2016, ■ the strains isolated in 2017; ● vaccine strains
Discussion
Our study has shown that EV-A71 vaccines were effective against medically-attended EV-A71-HFMD for children under five years of age, with a total VE of 83.7% for the recommended two doses of the vaccine, assessed using a test-negative design study. Our VE estimate was lower than the efficacy shown in the Phase III clinical trials for each EV-A71 vaccine,Citation10-Citation12 likely reflecting the difference between the real-world effectiveness and clinical-trial efficacy. In the Phase III trials, participants were carefully screened to assure enrollment of healthy subjects, evenly distributed into study arms. Our observational study tested effectiveness of the vaccines as they were used in Beijing.
Our VE estimates for children 6 to 35 months of age were slightly lower than for 36-to-59-month-old children, although not statistically-significantly different. The trend was consistent with a study on the immunogenicity of an EV-A71 vaccine comparing immune responses of children 36–59 months of age with children 6 to 35 months of age. Older children in this study showed a higher geometric mean titer (GMT) and a higher seroconversion rate than for children 6–35 months old.Citation17
Although the TND is less prone to bias than a traditional case-control study, selection of the control group has potential to influence VE estimates. To understand the robustness of our VE estimates, we used three types of control groups in our study: (1) all EV-A71-negative HFMDs, including HFMDs who tested negative for all enteroviruses and positive for other enteroviruses than EV-A71 (EV-A71-negative); (2) HFMDs who tested positive for other enteroviruses than EV-A71(other enteroviruses positive [OEV-positive]); and (3) HFMDs who tested negative for all enteroviruses (pan-EV-negative). When using EV-A71-negatives and pan-EV-negatives as controls, cases could be misclassified as controls due to imperfect laboratory testing, suboptimal swab quality, or long intervals between symptom onset and diagnosis.Citation18 Misclassification has potential to underestimate EV-A71 VE.Citation19 In this situation, the OEV-positive controls can produce more accurate EV-A71VE estimates. EV-A71 and other enteroviruses in our study were tested by Real Time PCR Kit manufactured by Jiangsu BioPerfectus Technologies Co., Ltd. The kit showed a sensitivity of ≥97.6% and a specificity of ≥99.7%. In addition, about 75% of the samples were obtained within 3 days of illness onset, and there was no significant difference in vaccination and diseases status by time between illness onset and sampling. To assure the sampling quality, clinicians were trained annually to collect specimens. In the sensitivity analysis, we found that our VE results were insensitive to control group definition, varying by less than 4 percentage points when two different restrictions on control group assignment were applied. Our finding was similar to that observed by Feng and colleagues in a meta-analysis of test-negative-design studies of influenza VE that used similar alternative control group restrictions to ours.Citation20 Therefore, we believe that using EV-A71-negative controls can producea reliable EV-A71 VE estimate as those with OEV-positive and pan-EV-negative controls if misclassification bias is minor. And it may be more resource-saving when the number of probable cases tested for EV-A71 and other enteroviruses is limited since it includes more participants than using OEV-positive and pan-EV-negative controls.
The test-negative design has been extensively used in recent years as a preferred method for estimating influenza and rotavirus VE in observational studies. However, potential bias may still exist. Besides misclassification bias mentioned above, confounding bias may exist. Calendar time was a potential confounding factor. Seasonality is correlated with both vaccine uptake and with incidence of non-EV-A71-HFMD, similar to the correlation of influenza season with influenza vaccination season. Younger cases (aged 6–35 months) were more likely to have been vaccinated and to have been infected with EV-A71. These potentially confounding factors should be adjusted by calendar month,Citation21 which is why we used sex, age and calendar month in our adjusted analyses to minimize bias of VE estimates.
There are some limitations to be considered in interpretation and generalization of our results. Health status, education, and socioeconomic level of guardians were not recorded in routine surveillance of HFMD, possibly influencing VE estimates since these factors may be associated with the likelihood of being vaccinated and the likelihood of developing non-EV-A71 induced HFMD. The power of our study is limited by sample size, precluding our ability to identify VE differences by age group. Our study was not designed to measure indirect effect (herd immunity) of EV-A71 vaccine, and as a short-term study, it could not address duration of protection. In addition, several studies on influenza VE suggest that vaccinated individuals infected with influenza may have milder symptoms than unvaccinated individuals,Citation22-Citation24 and therefore may be less likely to seek medical care. If there are large differences in vaccination rate between medically-attended cases and those without medical attention, our EV-A71 VE estimates may not represent those without medical attention which may limit the generalization of our findings.
In summary, the effectiveness of the inactivated monovalent EV-A71 vaccines was excellent. EV-A71 vaccine, as it is used in actual practice, is highly effective at protecting children under the age of five from medically-attended EV-A71-HFMD.
Methods
HFMD surveillance
HFMD case-based surveillance and virological surveillance have been performed in Beijing since 2007. In 2008, probable HFMD cases began to be reported in the electronic National Notifiable Infectious Diseases Reporting Information System (NNIDRIS),Citation25 capturing demographic information, date of symptoms onset, date of diagnosis, date of death (if applicable), disease severity, and clinical symptoms.Citation26 In HFMD virological surveillance, trained clinicians collected specimens from probable HFMD cases visiting outpatient departments of 29 sentinel hospitals in 16 districts of Beijing. Throat and/or rectal swabs were collected from probable HFMD cases sequentially, starting on the first day of each month until at least 5 mild probable HFMD cases were sampled per hospital. In addition to the throat or rectal swabs we obtained, fecal samples, vesicular fluid, and cerebrospinal fluid sampleswere collected from all the probable HFMD cases that met the definition of being severe or fatal HFMD cases, when possible.Citation7,Citation26 EV-A71, CV-A16 and CV-A6have been routinely tested for in Beijingsince 2013. A probable HFMD case was defined as a person clinically diagnosed with HFMD, with a papular or vesicular rash on the hands, feet, mouth, or buttocks, with or without fever.Probable HFMD cases with neurological complications (aseptic meningitis, encephalitis, encephalomyelitis, acute flaccid paralysis, or autonomic nervous system dysregulation) and/or cardiopulmonary complications (pulmonary edema, pulmonary hemorrhage, or cardiorespiratory failure) were classified as severe cases.Citation7
A confirmed EV case was defined as a probable case that tested positive for an enterovirus (EV-A71, CV-A16, CV-A6, or other non-EV-A71, non-CV-A16, and non-CV-A6 enteroviruses). All samples were analyzed with one-step, real-time reverse transcriptase polymerase chain reaction (RT-PCR),Citation16carried out with a pan-enterovirus detection kit, an EV-A71 detection kit, a CV-A16 detection kit, and a CV-A6 detection kit (Jiangsu BioPerfectus Technologies Co,, Ltd, CHN) according to the manufacturer’s instructions.
Study design
Use of EV-A71 vaccine started in Beijing on August 1, 2016. Since an immune response is elicited 28 days after the second dose,Citation10-Citation12 and since during the initial implementation of a new vaccination program coverage starts out low, medically -attended HFMD cases who were enrolled in the Beijing HFMD virological surveillance system in 2017 were recruited into this study. HFMD Cases born between January 1, 2012 and April 30, 2017 were included unless they (1) were born outside the predefined period (between January 1, 2012 and April 30, 2017); (2) were age <6 months when clinically diagnosed with HFMD [becausethese children could not have had an opportunity to receive EV-A71 vaccine before getting infected]; (3) had ever tested positive for EV-A71 before receiving any dose of EV-A71 vaccine; (4) received a second dose of EV-A71 vaccine less than 28 days before illness onset; (5) resided in Beijing less than 6 months; or (6) did not have an official vaccination record.
A test-negative design was used to estimate VE against EV-A71-HFMD. Cases were medically-attended HFMD cases who tested positive for EV-A71, while controls were medically-attended HFMD cases who tested negative for EV-A71. Participants who received only one dose or two doses of EV-A71 vaccine at least 28 days before illness onset were considered as vaccinated. Participants with other vaccination statuses were considered as unvaccinated.
In the main analysis, controls included children who were positive for an enterovirus other than EV-A71 and children who were negative for all enteroviruses – in other words, all subjects testing negative for EV-A71 infection.
We assessed sensitivity of VE estimates to control group inclusion criteria. Alternative control groups tested were (1) restriction of controls to subjects that tested positive for an enterovirus other than EV-A71, and (2) restriction of control to subjects testing negative for all enteroviruses (pan-EV-negative).
Data collection
Health care workers interviewed parents of all HFMD cases using a standardized questionnaire and entered data online in NNIDRIS within 24 hours of diagnosis. Trained clinicians collected throat swabs, rectal swabs, fecal samples, vesicular fluid, or cerebrospinal fluid from probable HFMD cases. Questionnaires were sent with specimens to the appropriate district CDC laboratory and were extracted and aggregated by Beijing CDC weekly.
EV-A71 vaccination records were obtained from BMSIIP, which has recorded vaccination information in Beijing continuously since 2009.Citation27,Citation28 BMSIIP covers all residents and migrant children who have been living in Beijing at least 6 months. Vaccination history and basic demographic information (name, sex, and birth date) were abstracted from BMSIIP for subjects receiving any vaccination in Beijing.
Virus characterization
To assess vaccine-virus match at the genetic level, a random sample of EV-A71 isolates obtained in 2016 and 2017 was characterized by sequencing their VP1 genes. Nucleotide sequences were assembled and aligned by MEGA software (ver.6.0.4) (Sudhir Kumar, Arizona State University, Arizona, USA). Neighbor-joining (NJ) phylogeny trees were built using 1,000 bootstrap replications.
Statistical analyses
We calculated VE for mild and for severe cases separately. We determined VE among mild cases by the number of dosesand by age group. Unconditional univariate and multivariate logistic regression models were used to estimate crude and adjusted odds ratios (OR). In the main analysis, ORs were calculated as the odds of EV-A71 vaccination among cases divided by the odds of EV-A71 vaccination among controls. VE was estimated as 100% × (1-OR). VE estimates were adjusted for sex, age group (6–35 months, 36–59 months), and calendar monthsusing unconditional multivariate logistic regression. We repeated the regression analyses with two alternative control groups. The estimates of VE by age group were not further stratified due to sample size limitations.Statistical analyses were performed with SPSS 20.0 software (SPSS Inc., Chicago, IL, USA). Percentages were calculated for categorical variables. All statistical tests were two-sided, and statistical significance was defined as p < 0.05.
Disclosure of potential conflicts of interest
No potential conflict of interest were disclosed.
Ethical review
The study was approved by the Institutional Review Board and Human Research Ethics Committee of the Beijing Center for Disease Control and Prevention.
Acknowledgments
The authors give special thanks to Dr. Lance E Rodewald (Chinese Center for Disease Control and Prevention) for his help in revising and polishing this manuscript.
Additional information
Funding
Related Research Data
References
- Podin Y, Gias EL, Ong F, Leong YW, Yee SF, Yusof MA, Perera D, Teo B, Wee TY, Yao SC, et al. Sentinel surveillance for human enterovirus 71 in Sarawak, Malaysia: lessons from the first 7 years. BMC Public Health. 2006;6:180. doi:10.1186/1471-2458-6-180.
- Chan LG, Parashar UD, Lye MS, Ong FG, Zaki SR, Alexander JP, Ho KK, Han LL, Pallansch MA, Suleiman AB, et al. Deaths of children during an outbreak of hand, foot, and mouth disease in Sarawak, Malaysia: clinical and pathological characteristics of the disease. For the outbreak study group. Clin Infect Dis. 2000;31:678–83. doi:10.1086/314032.
- Fujimoto T, Chikahira M, Yoshida S, Ebira H, Hasegawa A, Totsuka A, Nishio O. Outbreak of central nervous system disease associated with hand, foot, and mouth disease in Japan during the summer of 2000: detection and molecular epidemiology of enterovirus 71. Microbiol Immunol. 2002;46:621–27.
- Chan KP, Goh KT, Chong CY, Teo ES, Lau G, Ling AE. Epidemic hand, foot and mouth disease caused by human enterovirus 71, Singapore. Emerg Infect Dis. 2003;9:78–85. doi:10.3201/eid0901.020112.
- Van Tu P, Thao NTT, Perera D, Truong KH, Tien NTK, Thuong TC, How OM, Cardosa MJ, McMinn PC. Epidemiologic and virologic investigation of hand, foot, and mouth disease, southern Vietnam, 2005. Emerg Infect Dis. 2007;13:1733–41. doi:10.3201/eid1311.070632.
- National Health Commission of the People’s Republic of China. National surveillance of epidemic situation of notifiable communicable diseases in 2017 in China. Reviewed Jan 18 [accessed 2018 July 6]. http://www.nhfpc.gov.cn/jkj/s3578/201801/178264d9c8ab4d439a34284a4c84fee7.shtml.
- Xing W, Liao Q, Viboud C, Zhang J, Sun J, Wu JT, Chang Z, Liu F, Fang VJ, Zheng Y, et al. Hand, foot, and mouth disease in China, 2008–12: an epidemiological study. Lancet Infect Dis. 2014;14:308–18. doi:10.1016/S1473-3099(13)70342-6.
- Zheng Y, Jit M, Wu JT, Yang J, Leung K, Liao Q, Yu H. Economic costs and health-related quality of life for hand, foot and mouth disease (HFMD) patients in China. PLoS One. 2017;12:e0184266. doi:10.1371/journal.pone.0184266.
- Zheng Y, Yang J. Estimation of social economic burden caused by fatal hand, foot and mouth disease cases in China, 2013–2015. Dis Surveillance. 2017;32:516–20.
- Li R, Liu L, Mo Z, Wang X, Xia J, Liang Z, Zhang Y, Li Y, Mao Q, Wang J, et al. An inactivated enterovirus 71 vaccine in healthy children. N Engl J Med. 2014;370:829–37. doi:10.1056/NEJMoa1303224.
- Zhu F, Xu W, Xia J, Liang Z, Liu Y, Zhang X, Tan X, Wang L, Mao Q, Wu J, et al. Efficacy, safety, and immunogenicity of an enterovirus 71 vaccine in China. N Engl J Med. 2014;370:818–28. doi:10.1056/NEJMoa1304923.
- Zhu FC, Meng FY, Li JX, Li XL, Mao QY, Tao H, Zhang YT, Yao X, Chu K, Chen QH, et al. Efficacy, safety, and immunology of an inactivated alum-adjuvant enterovirus 71 vaccine in children in China: a multicentre, randomised, double-blind, placebo-controlled, phase 3 trial. Lancet. 2013;381:2024–32. doi:10.1016/S0140-6736(13)61049-1.
- Boom JA, Tate JE, Sahni LC, Rench MA, Hull JJ, Gentsch JR, Patel MM, Baker CJ, Parashar UD. Effectiveness of pentavalent rotavirus vaccine in a large urban population in the United States. Pediatrics. 2010;125:e199–207. doi:10.1542/peds.2009-1021.
- De Serres G, Skowronski DM, Wu XW, Ambrose CS. The test-negative design: validity, accuracy and precision of vaccine efficacy estimates compared to the gold standard of randomised placebo-controlled clinical trials. Euro Surveill. 2013;18:20585. doi:10.2807/1560-7917.ES2013.18.37.20585.
- Foppa IM, Haber M, Ferdinands JM, Shay DK. The case test-negative design for studies of the effectiveness of influenza vaccine. Vaccine. 2013;31:3104–09. doi:10.1016/j.vaccine.2013.04.026.
- Li J, Lin C, Qu M, Li X, Gao Z, Zhang X, Liu Y, Huang Y, Wang X, Jia L, et al. Excretion of enterovirus 71 in persons infected with hand, foot and mouth disease. Virol J. 2013;10:31. doi:10.1186/1743-422X-10-31.
- Gu W, Zeng G, Hu YM, Hu YS, Zhang Y, Hu YL, Wang Y, Li JX, Zhu FC. A comparative analysis of immunogenicity and safety of an enterovirus 71 vaccine between children aged 3–5 years and infants aged 6–35 months. Expert Rev Vaccines. 2018;17:257–62. doi:10.1080/14760584.2018.1430572.
- Sullivan SG, Tchetgen EJ, Cowling BJ. Theoretical basis of the test-negative study design for assessment of influenza vaccine effectiveness. Am J Epidemiol. 2016;184:345–53. doi:10.1093/aje/kww064.
- Fukushima W, Hirota Y. Basic principles of test-negative design in evaluating influenza vaccine effectiveness. Vaccine. 2017;35:4796–800. doi:10.1016/j.vaccine.2017.07.003.
- Feng S, Cowling BJ, Kelly H, Sullivan SG. Estimating influenza vaccine effectiveness with the test-negative design using alternative control groups: a systematic review and meta-analysis. Am J Epidemiol. 2018;187:389–97. doi:10.1093/aje/kwx251.
- Jackson ML, Phillips CH, Benoit J, Kiniry E, Madziwa L, Nelson JC, Jackson LA. The impact of selection bias on vaccine effectiveness estimates from test-negative studies. Vaccine. 2018;36:751–57. doi:10.1016/j.vaccine.2017.12.022.
- Deiss RG, Arnold JC, Chen WJ, Echols S, Fairchok MP, Schofield C, Danaher PJ, McDonough E, Ridore M, Mor D, et al. Vaccine-associated reduction in symptom severity among patients with influenza A/H3N2 disease. Vaccine. 2015;33:7160–67. doi:10.1016/j.vaccine.2015.11.004.
- VanWormer JJ, Sundaram ME, Meece JK, Belongia EA. A cross-sectional analysis of symptom severity in adults with influenza and other acute respiratory illness in the outpatient setting. BMC Infect Dis. 2014;14:231. doi:10.1186/1471-2334-14-231.
- Foppa IM, Ferdinands JM, Chaves SS, Haber MJ, Reynolds SB, Flannery B, Fry AM. The case test-negative design for studies of the effectiveness of influenza vaccine in inpatient settings. Int J Epidemiol. 2016;45:2052–59. doi:10.1093/ije/dyw022.
- Yang W, Li Z, Lan Y, Wang J, Ma J, Jin L, Sun Q, Lv W, Lai S, Liao Y, et al. A nationwide web-based automated system for outbreak early detection and rapid response in China. Western Pac Surveill Response J. 2011;2:10–15.
- Wang X, Wu X, Jia L, Li X, Li J, Li S, Qian H, Wang Q. Estimating the number of hand, foot and mouth disease amongst children aged under-five in Beijing during 2012, based on a telephone survey of healthcare seeking behavior. BMC Infect Dis. 2014;14:437. doi:10.1186/1471-2334-14-437.
- Qin Y, Zhang Y, Wu P, Feng S, Zheng J, Yang P, Pan Y, Wang Q, Feng L, Pang X, et al. Influenza vaccine effectiveness in preventing hospitalization among Beijing residents in China, 2013–15. Vaccine. 2016;34:2329–33. doi:10.1016/j.vaccine.2016.03.068.
- Zhang Y, Wu P, Feng L, Yang P, Pan Y, Feng S, Qin Y, Zheng J, Puig-Barbera J, Muscatello D, et al. Influenza vaccine effectiveness against influenza-associated hospitalization in 2015/16 season, Beijing, China. Vaccine. 2017;35:3129–34. doi:10.1016/j.vaccine.2017.03.084.