http://orcid.org/0000-0002-0083-6583 http://orcid.org/0000-0002-0083-6583
ABSTRACT
A single dose of the measles-mumps-rubella (MMR) vaccine has been applied in routine immunizations for children in China; however, the Immunoglobulin G (IgG) antibody level of mumps in children from kindergarten to early school age with MMR vaccine immunization history has not been elucidated. This study aimed to describe the immunity profile of children from kindergarten to early school age to identify the susceptible population. In Jiangsu Province, a cross-sectional survey of 4- to 8-year-old children who were vaccinated with at least one dose of MMR vaccine was conducted. IgG antibody results were analyzed by employing both the Enzyme-Linked Immunosorbent Assay (ELISA) cutoff values and the mixture model. A total of 7436 eligible subjects were enrolled: 3386 subjects were in kindergarten, and 4450 were in primary school. The overall seroprevalence (75.4%, 95% CI: 74.4%-76.4%) and Geometric mean concentration (GMC, 201.4U/ml, 95% CI: 194.1–209.4) of mumps antibodies in 2016 were low. The seroprevalence of children in kindergarten (78.1%, 95% CI: 76.6%-79.4%) was significantly higher than that of children in primary school (73.2%, 95% CI: 71.2%-74.6%). The GMC was negatively correlated with the time of inoculation (F = 32.17, P = 0.002). The mixture model enables a more comprehensive understanding of serological results by investigating four levels of antibody response, suggesting that there is a small fraction of the population with waning immunity. Children in kindergarten and primary school whom had received one dose of MMR vaccine were at a higher risk of mumps infection, particularly the 7-year-old group in the central region. Therefore, the single-dose MMR vaccine schedule has a limited impact on mumps control and prevention, and a two-dose MMR vaccine schedule should be introduced.
Introduction
Mumps is a common infectious disease in children globally, particularly in developing countries. Mumps cases have been frequently reported in school-age children, aged 3–9 years.Citation1 Individuals without mumps-containing vaccine (MuV) immunizations are at a higher risk of mumps infection. In China, a routine single-dose measles-mumps-rubella (MMR) vaccine schedule has been offered to children aged 18–24 months since 2008. MMR vaccination coverage in children over 18 months has steadily increased year by year and has exceeded 95% each year since 2009.Citation2 However, mumps has not been as effectively controlled as previous studies have reported,Citation1,Citation2,Citation3 suggesting that the single-dose MMR vaccine may have a limited effect on the mumps epidemic. According to the mumps vaccine position paper by the WHO, high coverage (95%) of the two-dose MMR vaccines in each birth cohort could effectively prevent the spread of mumps.Citation4 Serological surveillance plays a crucial role in providing a direct immunity profile of the population.Citation5 In 2016, a serological survey on mumps in children from kindergarten to early school age with at least one dose of MMR vaccine was conducted.
The fixed cutoff values of commercial Enzyme-Linked Immunosorbent Assay (ELISAs) are commonly used to analyze quantitative serological data when conducting a serological survey.Citation6 However, although these cutoff values are appropriate for clinical notifications and laboratory confirmations, this approach cannot make a distinction between negative samples and those with low concentrations of specific antibodies when explaining the results of a population-based epidemiological study.Citation7,Citation8,Citation9,Citation10 In contrary, the mixture models can be used to describe and interpret the age-stratified distribution of serological quantitative results. The objective of this study was to apply both fixed cutoff and mixture model to characterize the age-stratified distribution of results to fully understand the immunity profile of mumps in children from kindergarten to early school age with at least one dose of MMR vaccine, which may be benefit confirming the optimal age for the second dose MMR vaccine.
Results
Serological survey
A total of 7436 eligible serum samples from 7436 participants included were analyzed for mumps Immunoglobulin G (IgG) antibodies, including 3975 males (1849 from kindergarten age of 4–5 years, 2126 from primary school age of 7–8 years) and 3461 females (1537 from kindergarten age of 4–5 years, 1924 from primary school age of 7–8 years).
The overall seroprevalence of mumps antibodies was only 75.4% (95% CI 74.4%-76.4%). The seroprevalence in females was significantly higher than that in males (78.0% vs. 73.1%, χ2 = 115.5, P < 0.001). The children aged 7 years had the lowest seroprevalence at 72.4%, which was significantly lower than that of the 4-, 5-, and 8-year-old groups (χ2 = 18.8, P < 0.001). The seroprevalence of children in kindergarten (78.1%, 95% CI: 76.6%-79.4%) was significantly higher than that of children in primary school (73.2%, 95% CI: 71.2%-74.6%). When analyzing the data by region, the seroprevalence of 7-year-old children in the central region was 66.9%, which was significantly lower than that in the south and north regions, but the 5-year-old group in the central area had the highest seroprevalence compared with that in other regions. The seroprevalence of children with two doses of MuV was significantly higher than that of children with one dose of MuV (χ2 = 34.8, P < 0.001); see .
Table 1. Seroprevalence of mumps in children with MMR vaccination using the ELISA fixed cutoff values.
The overall Geometric mean concentration (GMC) of mumps antibodies in 2016 was 201.4 U/ml (95% CI: 194.1–209.4 U/ml). Significant differences among sex, age, and immunization history were observed across the whole province. The GMCs were significantly higher in females than in males (t = -5.6, P < 0.001). The lowest GMCs in 7-year-old children were observed in the central region. The GMC after two doses of MuV was significantly higher than that after one dose of MuV (t = -7.4, P < 0.001); see .
Table 2. Geometric mean concentration of mumps antibodies by sex, age group and vaccine status.
In addition, GMCs were negatively correlated with the time of inoculation (F = 32.17, P = 0.002); see .
Table 3. Relationship between times and GMC in children administered one dose of MMR vaccine.
Mixture model
The distribution fitting results showed that the mixture model with 4 component distributions was the optimal model to describe the frequency distributions for mumps antibody concentration data (χ2 = D4-D5 = 12.2, P = 0.24; χ2 = D3-D4 = 162.7, P < 0.001). The proportion of each component based on the geographical group is shown in . Each component represents a reactivity level, of which the mean (μ) and standard deviation (σ) are as follows: μ1 = 0.2, σ1 = 2.9; μ2 = 51.3, σ2 = 2.2; μ3 = 138.1, σ3 = 1.4; and μ4 = 457.1, σ4 = 2.1. Component distribution “1” indicates that these samples have the lowest reactivity. Each subsequent component indicates an increased reactivity level, with component distribution “4” suggesting that these samples have the highest reactivity. shows the proportions of serum samples designated by the mixture models to each component distribution by age group and area group at the provincial level. shows a strong agreement between the proportions of seronegative participants, as estimated by fixed cutoff values, and the proportions of level1+level2 participants, as estimated by the mixture model.
Table 4. Proportion of samples designated by the mixture model to reactivity level 1–4 (%, 95% CI).
Figure 1. Estimated proportion of samples by the mixture model to each component distribution, stratified by age group (A) and by geographical region (B).
Figure 2. Proportion of samples negative for mumps antibody as estimated by commercial ELISA cutoff values and as estimated by the model (95% CI) to be in reactivity level 1 and level≤2, respectively.
Discussion
In recent years, several mumps outbreaks were reported in many countries, most of which occurred in children or adolescents who have received two-dose MMR vaccines.Citation11 These outbreaks triggered the studies of the optimal age for the second dose MMR vaccine.Citation12,Citation13 Completing two doses of MMR vaccines at an earlier age (such as before 24 months) may lead to a higher risk of mumps infection during teenage years.Citation11,Citation12 Therefore, the merit of this study was providing a comprehensive understanding of immunity profile to mumps in children aged 4–8 years with one dose MMR vaccine, which may benefit confirming the optimal age of the second MMR vaccine.
In this study, a cross-sectional analysis was conducted to understand the immunity profile of children who have received at least one dose of MMR vaccine in Jiangsu Province, China. Both ELISA fixed cutoff and mixture models were applied to analyze the serological data. The results of this study indicate that the overall seroprevalence (75.4%) to mumps virus was too low to interrupt mumps transmission.Citation14,Citation15 In addition, the seroprevalence of one dose of MMR vaccine was significantly lower than that of two doses of MMR vaccine (74.8% vs. 89.2%), suggesting that one dose of MMR vaccine may be insufficient in preventing and controlling the prevalence and outbreak of mumps. Furthermore, both the seroprevalence and GMC in kindergarten-age children were significantly higher than those in primary school-age children. A possible reason for this finding may be that the antibody concentration to mumps decreases gradually over time after vaccination.Citation16,Citation17 Studies have also shown that mumps cases are frequently reported in children aged 3–9 years.Citation18 Therefore, a second dose of MuV may be considered for children in kindergarten and primary school to avoid potential mumps epidemics.
Notably, both GMCs and seroprevalence were statistically lower in males than in females. Similar studies have also shown that higher proportions of cases involving males have been observed in mumps outbreaks.Citation15,Citation18,Citation19 A high contact rate among male students may decrease vaccine-induced protection and facilitate mumps transmission.Citation17,Citation20 The lowest seroprevalence was observed in 7-year-old children across the whole province, most notably in the central region, which exhibited a seroprevalence of only 66.9%. It's urged to take further action to close the immunization gap due to the high risk of mumps infection.
Traditionally, seroprevalence is calculated using a commercial ELISA fixed cutoff value. However, it is well-known that the fixed thresholds, which are primarily used in clinical diagnosis settings, typically have greater specificity than sensitivity and, therefore, tend to underestimate the prevalence in the target population.Citation21,Citation22 In the sero-survey, our goal was not to use the cutoff values to classify individual results as positive or negative but to investigate various levels of antibody response to vaccination or infection in a specific population. Therefore, we used a mixture model approach to fully describe the immunity profiles of children in Jiangsu Province who received the MuVs.
Mixture model analysis depends on the number of components to be analyzed and the distribution of each component. In this study, a normal distribution for each component was selected because logarithmic ELISA results generally follow a normal distribution.Citation7,Citation8 By comparing the deviance of a sequence of models with increasing complexity, an age-specific model with 4 components was selected to assess the serological data. During the construction of the mixture model, it was assumed that the mean and deviation of each component distribution was independent of age; correspondingly, we can describe the distribution of decaying antibody levels after vaccination using several components. Antibody levels produced in response to vaccination are attributed to many factors. It would be reasonable to select 3 component distributions to represent negative, weak positive, and strong positive. However, an additional component was introduced to enable the model to describe the possible decay or lower levels in antibody concentrations after vaccination. Therefore, the distribution (reactivity level) with the lowest mean could represent susceptible individuals, while other distributions with reactivity levels≥2 could represent a higher level of reactivity.Citation9,Citation10
Level 1 shows the lowest reactivity in serum with almost no detectable antibodies. Approximately 2% of the population in each age group had almost no detected antibodies. This finding may be interpreted as the failure to induce antibodies to mumps with the single-dose of MMR vaccine in China.Citation14 These populations are the focus of surveillance, and booster immunizations should be offered to these children immediately.
Level 2 can be considered a negative level with a very small number of weak-positive individuals, in whom a low antibody titer can be detected, but not at a sufficient level to protect the individual from infection. Significant differences in the number of level 2 individuals among the four age groups were observed, with the highest proportion in the 7-year-old age group. Over time, the proportion of individuals classified as level 2 may gradually increase due to waning immunity.Citation23 Thus, additional measures should be taken to interrupt this trend and avoid outbreaks of mumps.
Level 3 can be considered a positive level with a lower antibody titer than level 4. This level can protect the body from infection to a certain extent and was mostly observed in the 7-year-old group in the central region. Individuals in this level could be converted to level 4 if a second dose of MMR vaccine is introduced into the routine immunization schedule.
Level 4 can be assumed to correspond to a robust immune response that surpasses the level of protection, which can protect children from infecting with mumps. A number of people classified as level 4 have received two doses of the MuV. As seen in , level 4 exhibits a greater difference in the distribution among area groups, with the highest proportion observed in the northern region. In addition, the proportion of individuals in the highest reactivity level does not show a rising or declining trend with age.
There are limitations in this study. Firstly, only the children with 4–8 years old were selected and the level of antibodies before receiving the first dose MMR vaccine was unclear. Secondly, to avoid the overestimation of the immunity of children with one dose MMR, children with mumps infection has been excluded, but this exclusion criterion may also make the results biased because a certain proportion of children who may still infect with mumps after received one dose MMR vaccine were also excluded.
In conclusion, both the GMC and seroprevalence in children at school age were low. And the single-dose MMR vaccine strategy may have a limited impact on the control of mumps, and the two-dose MMR vaccines schedule should thus be introduced. Children in kindergarten and primary school immunized with one dose of MMR vaccine were at a high risk of mumps infection, particularly the 7-year-old group in the central region.
Materials and methods
Serological survey
A cross-sectional survey was conducted using a multistage stratified random sampling method in 2016. The whole province was divided into three regions based on the socioeconomic status and the level of implementation of the immunization program: south, central and north. 3 counties in the north region, 2 counties in the south region, and 1 county in the central region were chosen using a random number –5 primary schools and 3–5 kindergarten classes in each county were randomly selected. All the children in the youngest class of each kindergarten and first grade class were recruited. The inclusion criteria were as follows: (1) aged 4–5 years in kindergarten or aged 7–8 years in primary school; (2) local residents for at least 3 months; (3) good physical health(with axillary temperature < 38℃, and without acute or chronic diseases); and (4) documented as being vaccinated with at least one dose of MuV. The exclusion criteria were as follows: (1) did not receive MuV; (2) refused collection of venous blood; (3) had been infected with mumps; and (4) has serious illness or other medical reasons for not participating in the study after clinical evaluation.
A total of 7436 serum samples from 7436 participants were collected from January 2016 to November 2016. The study was carried out in accordance to the Declaration of Helsinki and Good Clinical Practice guidelines and was approved by the Medical Ethics Committee of the Jiangsu Provincial Center for Disease Control and Prevention(NO: SL2015-B015–02). Parents of children who provided serum samples were requested to sign a written informed consent and to fill out a questionnaire regarding their children's information and immunization history, such as age, gender, date of birth, date of vaccination, and date of sampling. MuV vaccination status was based on the record of vaccination certificate or vaccination records from the Jiangsu Provincial Children Vaccination Information System. This study has been registered at www.clinical trials.gov under NCT02901990.
Laboratory assay
The serum samples were processed and frozen at -70°C until testing. Detection of mumps-specific IgG antibodies was performed using ELISA at the laboratory of the Department of Expanded Program on Immunization at the Jiangsu Provincial Center for Disease Control and Prevention. To avoid test bias, all detection tests were conducted by the same staff members using commercial ELISA kits from Institut Virion\Serion GmbH (SERION ELISA classic anti-mumps virus IgG, batch number: SLF.CL). The ELISA results were firstly expressed as optical density (OD) measurements at 402 nm, and later converted to the antibody concentration values (U/ml), using software from SERIO according to the product instructions. The quantitative results of ELISA were categorized as positive with concentrations ≥ 108 U/ml or negative with concentrations < 108U/ml. Equivocal results were retested and categorized as a positive or negative finding using the same methods.
Statistical analysis
The GMCs of mumps antibodies and seroprevalence with 95% CIs stratified by gender, age and geographical region were calculated. Comparisons of seroprevalence in different stratifications were performed using the χ2 test. Analysis of variance was used to compare the GMCs in different groups, and the Student-Newman-Keuls q test was used for multiple comparisons among different age groups. The Cochran-Mantel-Haenszel χ2 test was used to analyze the differences between groups. Bonferroni correction methods were used for multiple comparisons of seroprevalences if significant differences were observed across regions and ages. Linear correlation was used to analyze the relationship between GMCs and the time since inoculation. All statistical tests used two-sided α = 0.05as the significance threshold and were performed using R 2.10.0 statistical software.Citation24
Mixture model
After serological data were analyzed using the fixed cutoff value, a mixture model was applied to the antibody concentration analysis as described by previous studies.Citation8 The mixture model assumes that antibody concentrations detected from sero-samples from participants represent different immunity statuses, which can be described by distribution. The simplest model consisted of two immunity statuses, corresponding to two distributions: positive and negative. If the parameters of such distributions (for example, the mean and standard deviation for a normal distribution) and the proportions of samples with each status are known, the overall distribution of antibody results may be determined. The mixture model takes an inverse approach by fitting the overall distribution of the results and then estimating the distribution parameters and the proportion of samples for each immunity status. A nested mixture model was used to estimate two or more distributions stratified by age.
Individual results were aggregated by 33 reactivity categories of log (antibody concentration) values stratified by geographical region and age group. A χ2 test was used to compare the deviance of each increasingly complex, nested model to that of its immediate simpler predecessor, thereby enabling the simplest fit with the number of component distributions.Citation8,Citation9 For a given component, it is assumed that the distribution of results is age-independent and follows a normal distribution. Age-related changes in reactivity are reflected in the model by changes in the proportions and attributed to each component distribution. A maximum-likelihood technique was used to estimate the parameters. Parameters of mean and standard deviation were used to estimate each of the n reactivity level distributions, and subsequent n-1 parameters were included to describe the proportions in each reactivity level for each age group.Citation6,Citation7 Details of the parameter estimation and goodness of fit are provided in the appendix.
Disclosure of potential conflicts of interest
No potential conflicts of interest were disclosed.
Supplemental Material
Download MS Word (23.6 KB)Acknowledgments
We would like to thank all of the following CDCs in Jiangsu Province for their field studies: Changzhou Center for Disease Control and Prevention (CDC), Huaian CDC, Lianyungang CDC, Taizhou CDC, Zhenjiang CDC, Xuzhou CDC, Wujin district CDC, Lianshui county CDC, Ganyu district CDC, Gaogang district CDC, Tongshan district CDC, and Danyang county CDC.
Additional information
Funding
References
- Liu Y, Hu Y, Deng X, Wang Z, Lu P, Ma F, Zhou M, Liu P, Min, J. Sero-epidemiology of mumps in the general population of Jiangsu province, China after introduction of a one-dose measles-mumps-rubella vaccine. Sci Rep. 2015;5:14660. doi:10.1038/srep14660. PMID: 26423223.
- Liu Y, Liu Z, Deng X, Hu Y, Wang Z, Lu P, Guo H, Sun X, Xu Y, Tang F, Zhu FC. Waning immunity of one-dose measles-mumps-rubella vaccine to mumps in children from kindergarten to early school age: A prospective study. Expert Rev Vaccines. 2018:1–8. doi:10.1080/14760584.2018.1445529. PMID:29478347.
- Ma C, Liu Y, Tang J, Jia H, Qin W, Su Y, Wang H, Hao L. Assessment of mumps-containing vaccine effectiveness during an outbreak: Importance to introduce the 2-dose schedule for China. Hum Vaccin Immunother. 2018;29:1–6. doi: 10.1080/21645515.2018.1428508. PMID:29377772.
- WHO. Mumps vaccines WHO position paper – 2007 Weekly epidemiological record. Available from [ Accessed 2017 Oct 21]. http://www.who.int/wer/2007/wer8207.pdf?ua=1
- Shah M, Quinlisk P, Weigel A, Riley J, James L, Patterson J. Mumps outbreak in a highly vaccinated university-affiliated setting before and after a measles-mumps-rubella (MMR) vaccination campaign-Iowa, July 2015-May 2016. Clin Infect Dis. 2018;66(1):81–88. doi:10.1093/cid/cix718. PMID:29020324.
- Liu Y, Tao H, Ma F, Lu P, Hu Y, Ding X, Liu P, Min J, Li J, Xiao S. Sero-epidemiology of measles in general population in Jiangsu province of China: application of mixture models to interpret the results from a cross-sectional study. Vaccine. 2011;29(5):1000–4. doi:10.1016/j.vaccine.2010.11.081. PMID:21163248.
- Tong DDM, Buxser S, Vidmar TJ. Application of a mixture model for determining the cutoff threshold for activity in high-throughput screening. Comput Stat Data Anal. 2007;51(8):4002–12. doi:10.1016/j.csda.2006.12.014.
- Rota MC, Massari M, Gabutti G, Guido M, De Donno A, Ciofi degli Atti ML. Measles serological survey in the Italian population: interpretation of results using mixture model. Vaccine. 2008;26(34):4403–9. doi:10.1016/j.vaccine.2008.05.094. PMID:18585420.
- Gay NJ. Analysis of serological surveys using mixture models: application to a survey of parvovirus B19. Stat med. 1996;15(14):1567–73. doi:10.1002/(SICI)1097-0258(19960730)15:14<1567::AID-SIM289>3.0.CO;2-G. PMID:8855482.
- Vyse AJ, Gay NJ, Hesketh LM. Interpreting serological surveys using mixture models: the seroepidemiology of measles, mumps and rubella in England and Wales at the beginning of the 21st century. Epidemiol Infect. 2006;134(6):1303–12. doi:10.1017/S0950268806006340. PMID:16650326.
- Hamami D, Cameron R, Pollock KG, Shankland C. Waning Immunity Is Associated with Periodic Large Outbreaks of Mumps: A Mathematical Modeling Study of Scottish Data. Front Physiol. 2017;5(8):233.doi: 10.3389/fphys.2017.00233. PMID:28487657.
- Smetana J, Chlibek R, Hanovcova I, Sosovickova R, Smetanova L, Smetana J, Chlibek R, Hanovcova I, Polcarova P, Gal P, et al. Serological Survey of Mumps Antibodies in Adults in the Czech Republic and the Need for Changes to the Vaccination Strategy. Hum Vaccin Immunother. 2018;14(4):887–93. doi:10.1080/21645515.2017.1412021. PMID:29206078.
- Gu XX, Plotkin SA, Edwards KM, Sette A, Levy O, Sant AJ, Mo A, Alexander W, Lu KT, Taylor CE, et al. Waning immunity and microbial vaccines – workshop of the national institute of allergy and infectious diseases. Clin Vaccine Immunol. 2017;24(7):e00034–17. doi:10.1128/CVI.00034-17. PMID:28490424.
- Liu YB, Hu Y, Deng XY, Wang ZG, Sun X, Lu PS, Guo HX, Tang FY, Zhou MH. Study of mumps immunity after administrating measles-mumps-rubella vaccine among children aged 2–7 years old in Jiangsu Province in 2015. Zhonghua yu fang yi xue za zhi. 2017;51(7):593–7. doi: 10.3760/cma.j.issn. PMID:28693082.
- Diaz Ortega JL, Castaneda D, Arellano Quintanilla DM, Martinez D, Trumbo SP, Fernandez de Castro J. Antibody persistence in children aged 6–7years one year following booster immunization with two MMR vaccines applied by aerosol or by injection. Vaccine. 2017;35(23):3116–22. doi:10.1016/j.vaccine.2017.04.027. PMID:28457672.
- Gu XX, Plotkin SA, Edwards KM, Sette A, Mills KHG, Levy O, Sant AJ, Mo A, Lu KT, Taylor CE, et al. Waning immunity and microbial vaccines – workshop of the national institute of allergy and infectious diseases. Clin Vaccine Immunol. 2017;24(7):10. doi: 10.1128/cvi.00034-17. PMID:28490424.
- Urbano PR, Fujita DM, Romano CM. Reemergence of mumps in Sao Paulo, Brazil – the urgent need for booster shot campaign to prevent a serious infectious disease. Rev Soc Bras Med Trop. 2017;50(4):535–8. doi:10.1590/0037-8682-0320-2016. PMID:28954076.
- Wang H, Hu Y, Zhang G, Zheng J, Li L, An Z. Meta-analysis of vaccine effectiveness of mumps-containing vaccine under different immunization strategies in China. Vaccine. 2014;32(37):4806–12. doi:10.1016/j.vaccine.2014.05.061. PMID:25000591.
- Leuridan E, Maertens K, Wautier M, Hutse V, Theeten H. Susceptibility to measles, mumps, and rubella in 5-year-old children in Flanders, Belgium. Eur J Pediatr. 2015;174(7):925–32. doi:10.1007/s00431-014-2481-5. PMID:25579232.
- Demicheli V, Rivetti A, Debalini MG, Di Pietrantonj C. Vaccines for measles, mumps and rubella in children. Cochrane Database Syst Rev. 2012;(2):CD004407. doi: 10.1002/14651858.CD004407.pub3. PMID:22336803.
- Vyse AJ, Gay NJ, Hesketh LM. Interpreting serological surveys using mixture models: the sero-epidemiology of measles, mumps and rubella in England and Wales at the beginning of the 21st century. Epidemiol Infect. 2006;134(6):1303–12. doi:10.1017/S0950268806006340. PMID:16650326.
- Parker RA, Erdman DD, Anderson LJ. Use of mixture models in determining laboratory criterion for identification of seropositive individuals: application to parvovirus B19 serology. J Virol Methods. 1990;27(2):135–44. doi:10.1016/0166-0934(90)90130-8. PMID:2156877.
- Bockelman C, Frawley TC, Long B, Koyfman A. Mumps: An Emergency Medicine-Focused Update. J Emerg Med. 2018;54(2):207–14 doi:10.1016/j.jemermed.2017.08.037. PMID:29110978.
- A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL [ Accessed 2017 Oct 21].https://www.R-project.org/