abstract
Carbapenem-resistant Enterobacteriaceae (CRE) are an emerging and troublesome group of pathogens. Risk factor studies, outcome studies, and randomized trials are three types of studies conducted to answer different types of questions regarding CRE. These studies pose different types of challenges. We discuss issues in the design and analyses of case-control studies, cohort studies, and randomized trials aimed to address various research questions regarding CRE.
Carbapenem-resistant Enterobacteriaceae (CRE) are an emerging and troublesome group of pathogens. The National Healthcare Safety Network (NHSN) reported that approximately 12% of all Klebsiella isolates recovered in 2009–2010Citation1 were carbapenem resistant, compared with slightly less than 1% in 2000. The most common Enterobacteriaceae exhibiting carbapenem resistance are K. pneumoniae followed by Enterobacter spp.Citation1 In 2013, the CDC assigned the highest threat level to CRE declaring they require urgent public health attention.Citation2
When an emerging pathogen such as CRE evolves, 3 types of studies are often done: a) risk factor studies; b) outcome studies; and c) randomized trials (). Risk factor studies are conducted to identify risk factors that will help lead to interventions that will decrease the emergence of the resistant pathogen. Risk factor studies may also help identify high-risk patients for CRE randomized trials. Outcome studies help delineate the public health burden of the resistant bacteria such as morbidity, mortality and costs. For both of these types of studies, the epidemiology and statistical methodology to achieve high validity of the studies is complex. As in many observational studies, there is never a perfect solution that yields a study design that is 100% internally valid. Randomized trials yield the highest level of causal evidence and often are the definitive study conducted to evaluate interventions to treat antibiotic-resistant bacteria.
Table 1. CRE Study Types.
Epidemiological principles applied to CRE to be discussed in this manuscript include: a) case-control study design; b) cohort study design; and c) the potential use of external controls instead of randomized controls for evaluating interventions to treat infections caused by CRE.
Case control studies for CRE
The case control study design is still chosen most often for risk factor studies for CRE. In these studies, CRE is the outcome and often the main exposure of interest is an individual antibiotic. The following are some important methodological issues that are relevant to CRE case control studies.
Importance of control group selection
The identification of the correct control group is often the toughest challenge in case control studies.Citation3,4 Appropriate control group selection is driven by the intricacies of the research question. Thus the research question requires careful thought and an appropriate level of specificity. The identification of the appropriate study base from which to select control patients is the primary challenge in the design of case-control studies. Control patients should be selected from the same source population or study cohort base that gives rise to the case patients during the same time periods from which the cases arise; by choosing controls using this method you avoid introducing a selection bias. Estimation of the relative risk (measured most often by odds ratios) in a case-control study relies on a comparison of the exposure frequency in case patients with control patients. If control patients are selected in a manner such that their frequency of exposure is not representative of the base population, then relative risk estimates will be biased.Citation5
For many CRE risk factor studies, the research question in the hospital setting is what are the risk factors for acquiring CRE among hospitalized patients? An investigator may be interested in the outcome of having a positive clinical culture for CRE. For this question, cases are patients with a positive clinical culture for CRE. Controls should be selected from the base population or cohort consisting of hospitalized patients in the same time period and same locations among which cases of CRE arose.Citation6,7 Thus, controls should be representative patients from these same locations and in the same time periods as cases. Controls should not be patients with carbapenem-susceptible Enterobacteriaceae (CSE). Patients with CSE do not represent the base because they are not the source population among which case patients arise. The choice of CSE patients would likely lead to biased estimates of relative risk because of a distorted estimate of exposure frequency in the source population; similar to what was found in other antibiotic-resistant studies.Citation5 If the CSE control group was chosen, then estimates of effect would be biased and antibiotics may be incorrectly identified as risk factors. For example, the odds ratios obtained for carbapenems would likely be too large because patients in the CSE control group will have such a low frequency of carbapenem use. There are less common instances where the CSE control group may be more appropriate: a) If the research question is “what are the risk factors for developing CRE among a cohort of patients with CSE;” b) If the research question is what empiric therapy should be chosen among a cohort of patients with Enterobacteriaceae infection.
If one wishes to contrast CRE patients to CSE patients and not introduce a selection bias, then a viable study design is the case-case control study design.Citation8 In this study design, 2 separate case-control analyses are done within a single study. The first analysis compares CRE patients to hospitalized control patients. The second analysis compares CSE patients to hospitalized control patients. This allows you to in the same manuscript to compare risk factors for CRE to risk factors for CSE. Ideally, the 2 control groups should not be identical since the control group for CRE may randomly have some CSE patients whereas the control group for CSE should not have any patients with CSE. These two case-control studies are then compared using a side-by-side qualitative comparison of risk factors identified. When the 2 case control studies are compared and contrasted, risk factors associated with CRE are contrasted with risk factors associated with CSE. The case-case control study design does not introduce the selection bias discussed above.
Importance of controlling for confounding variables
A confounding variable is a variable entangled with the exposure of interest and the outcome of interest that masks the true relationship between the exposure and the outcome.Citation3,9 Confounding variables must be measured and controlled for either in the study design (e.g. randomization, matching, restriction) or in the analysis (e.g., stratified or multivariable analysis). For risk factor studies, we generally recommend multivariable models to control for confounding. Matching has the disadvantage of not being able to analyze the matched variables and the requirement of a matched statistical analysis. If a matched statistical analysis is not performed when matching was performed in the design, an additional selection bias is introduced.Citation3,4 Numerous issues of model selection and model building are important and discussed in detail elsewhereCitation10,11 but will not be discussed here.
Some of the most important confounding variables for antibiotic-resistant bacteria risk factor studies include: a) antibiotics; b) comorbid conditions; and c) severity of illness.
Patients who acquire antibiotic-resistant bacteria such as CRE while in the hospital often receive numerous antibiotics prior to having their positive surveillance or clinical culture with CRE. These antibiotics are sometimes confounding variables and sometimes collinear variables. Confounding variables need to be controlled for in the statistical analysis as they can distort estimates of effects of interest. Collinear variables should not be controlled for in the statistical analysis. Collinear variables are highly correlated, meaning that one can be linearly predicted from the other with a substantial degree of accuracy
Comorbid conditions such as diabetes, cancer diagnoses, and immunocompromised states have been shown to be risk factors for antibiotic resistant bacteria.Citation12-14 These comorbid conditions are confounding variables because patients with these conditions are more likely to develop the outcome (CRE) and are more likely to have received an antibiotic that is a risk factor of interest. These comorbid conditions need to be adjusted for in order to get an unbiased estimate of the association between the risk factor of interest (exposure e.g. antibiotic) and the outcome of antibiotic resistance (CRE). Many attempt to adjust for comorbid conditions using a standardized score such as the Charlson comorbidity index or the Elixhauser comorbidity index or the Chronic Disease Score. A limitation of these scores is that they were originally developed using non-infectious disease datasets and thus some components lack biological plausibility to infectious disease outcomes and the scores need further infectious disease specific validation. More work is needed regarding the optimal methods of comorbidity adjustment specific to infectious diseases and antibiotic resistance.
Severity illness is another important confounding variable. An important point to clarify is that severity of illness is different from comorbidity and the terms should not be used interchangeably. For example, a patient could have no comorbid conditions but could have an extremely high severity of illness and be in septic shock due to Neisseria meningitis. This patient could then subsequently acquire a CRE. Severity of illness is a confounding variable because patients with high severity of illness are more likely to receive antibiotics (exposure of interest) and patients with high severity of illness are more likely to have breathing tubes, Foley catheters etc. that may lead to a higher probability of the outcome. At present, most of the better severity of illness adjustment scores have only been developed and validated for ICU use. In infectious diseases, the McCabe and Jackson score has been used often but has a number of limitations.Citation15,16 The timing of measuring severity of illness is important in that it should not be measured after the outcome has occurred. One must make sure that severity of illness is a confounding variable and not an intermediate variable and this point also affects the optimal timing of severity of illness measurement; an intermediate variable is a variable that lies between the exposure/study factor and the outcome in a causal chain. Future work is needed to improve severity of illness adjustment.
Numerous authors have written important references on the topic of case-control studies and antibiotic resistance.Citation7,17
Cohort studies for CRE
When an emerging pathogen such as CRE evolves, outcome studies are often done to demonstrate that CRE leads to an increase in poor patient outcomes. Outcomes most commonly studied include mortality and healthcare utilization such as cost and length of stay. The aim of these studies is often to demonstrate that a CRE infection leads to increased costs, increased length of stay and increased mortality. Appropriately, the cohort study design is used most often for these CRE outcome studies. Similar to what was discussed in case-control studies, there is no perfect study design to address the exact impact of CRE on mortality and length of stay.
Often unfortunately, risk factor studies and outcome studies are combined in the same paper and incorrectly referred to as a case-control study often with flawed methodology for both the risk factor study and the outcome study. In contrast to the risk factor case control study where acquiring a CRE is the outcome, in these cohort outcome studies CRE is the exposure. The outcome is often length of hospital stay, hospital costs or mortality. The non-exposed group is patients who do not have CRE. One of the most difficult issues in these types of cohort designs is the choice of the non-exposed category. Researchers have struggled with the optimal non exposed group that allows proper control for confounding variables such as comorbidity and severity of illness.
The study designs chosen most often to assess outcomes of antibiotic-resistant bacteria are the matched cohort design or the cohort design with multivariable analysis to control for confounding. The matched cohort design is often chosen to try to control for confounding variables that could cause an increase in length of stay or mortality and are associated with patients who have a CRE. Possible matching variables can include age, immune status, comorbidity and severity of illness prior to infection with the antibiotic-resistant bacteria. However, numerous articles have outlined methodological issues related to the matched cohort design. Two methodological issues we want to highlight are lead time bias and competing risk.
Lead time bias
Lead time bias arises because acquiring a CRE is a time-dependent event, i.e., CRE status may change over the course of observation. Often a patient does not have CRE at hospital admission but acquires CRE while in the hospital. The impact of CRE on the outcome of length of stay will be distorted unless the event of CRE is modeled as time dependent. When modeling CRE as time-dependent one must carefully define “time-zero.” If CRE status was periodically evaluated and CRE was acquired at time t, then a binary variable could be created such that for times before t there was no CRE but for times after t, CRE was present. This variable could then be used as the exposure in a regression model to obtain an estimate of the association between CRE and length of stay.Citation10
Often outcome studies of antibiotic-resistant bacteria falsely assume and model the exposure as if it occurs on hospital admission. If modeled this way, the estimates of interest will suffer from time-dependent bias. For length of stay and cost outcomes, not adjusting for lead time bias will lead to an overestimate of the effect of CRE. If using the matched cohort study, an improved method involves controlling for this time to CRE infection in the exposed and non-exposed. This has been described in numerous non-infectious disease studies.Citation18 It has been described in detail in infectious diseases relative to the study of healthcare-associated infections and outcomes.Citation19,20
Competing risk
CRE studies may often want to study the effect of CRE on patient mortality and on hospital costs including length of stay. Epidemiologically, these outcomes are called competing risks. For example, suppose that CRE infections lead to an increased death rate and shortened survival. This would lead to patients having shorter lengths of stay and smaller health care costs due to earlier death times. Separate matched cohort analyses or separate outcome analyses would falsely lead to the incorrect conclusion that CRE is protective and leads to shorter hospital utilization costs. When outcomes of interest are competing, more sophisticated analyses should be done including Cox regression analyses and multi-state models in order to reach the proper conclusions.Citation21,22
Possible advanced methods that may solve some of these problems and are being studied include the use of instrumental variables and the use of multi-state models.Citation23
Can we use external controls in the CRE setting?
More effective antibiotics are needed to treat CRE infections due to the decreased efficacy of currently available antibiotics and the increasing prevalence of CRE. But despite the increasing prevalence of CRE, identifying and enrolling participants into clinical trials that evaluate new interventions for the treatment of CRE can be challenging. Thus researchers might consider alternative designs that require fewer trial participants and can produce results more quickly but still maintain scientific validity. One design option for consideration is the use of externally-controlled trials (ECTs). ECTs may allow for a reduction of the necessary number of prospectively-identified trial participants, thus easing recruitment burden, reducing costs, and resulting in more timely trial completion relative to randomized controlled trials (RCTs). Can we use external controls in the CRE setting?
ECTs can be historical-controlled trials (HCTs) where control group data are collected retrospectively, or concurrent external controlled trials (CECTs), where control group data is collected concurrently. During the analysis of ECTs, the external control is compared with the prospectively enrolled treatment arm with respect to important endpoints of interest.
An advantage of using ECTs is that the trial will require fewer prospectively enrolled participants due to the absence of a prospective control group, thus providing resource and time efficiency. For this reason, ECTs are often considered in trials when the eligible patient pool is limited as would be expected with CRE. ECTs can be more attractive for prospective patients since they know what treatment will be assigned in contrast to RCTs. Patients in RCTs do not know what treatment they will received and may remain blinded until the trial is over.
The major drawback of ECTs is that they are non-randomized studies. Randomization ensures the expectation of between-arm balance with respect to all factors, known or unknown, measured or unmeasured. Randomization provides the theoretical foundation for valid estimation of treatment effects. Without randomization, estimates of treatment effects can be biased. Because they are non-randomized studies, ECTs are potentially vulnerable to the biases of observational studies.
Bias can occur if the controls systematically differ from the prospective participant group with regard to important factors in a manner that can affect outcome. Differences may occur due to participant selection (e.g., patients with more favorable prognosis or lesser disease severity are selected for the prospective component of the trial but poor risk patients are excluded), supportive care, concomitant therapies, follow-up strategies, and outcome evaluation methods. It may also be challenging to define a “time zero” representing a baseline in the external component of the ECT, to align with “time 0” in the prospective component of the trial. If the ECT is an HCT, then bias may occur due to factors that have changed since the time the historical control group data were collected (e.g., evolving resistance, improvement in medical practice and patient standard of care, or diagnostic criteria). Furthermore ECTs are not blinded and thus are subject to bias when eligibility or outcomes are assessed by clinicians or patients. In ECTs, clinicians may also selectively prescribe additional therapies given the knowledge or the treatment assignment and the seriousness of CRE infections.
ECTs have been used rarely in clinical trials for late-stage drug development due to the concerns for these potential biases. International guidanceCitation24 recommends reserving ECTs for specific situations where the effects of interventions are large, the natural history of the disease is well understood, and outcomes are not greatly affected by e.g., patient demographics. It is further recommended that ECTs be limited to cases in which the endpoints are objective (e.g. all-cause mortality) to avoid subjective evaluations given the unblinded nature of ECT trials.
Research suggests that ECTs tend to produce “positive” results more frequently than RCTs. Sacks et al. (1982)Citation25 reviewed 50 RCTs and 56 HCTs evaluating the same 6 therapies and found 79% of the HCTs but only 20% of the RCTs demonstrated superiority of the test group to the control.Citation25
An example in infectious disease setting is patulin, a metabolic product of Penicillium patulum Bainier. Patulin was studied for the treatment of the common cold in a non-randomized, double-blinded concurrent controlled (patulin in buffer vs. buffer alone) clinical trial of 180 subjects in 1943. The number of subjects that improved at 48 hours in the patulin in buffer arm was 55/95 (58%) vs. Eight/85 (9.4%) in buffer alone arm, a difference of 48%, 95% CI = (35%, 60%), p < 0.002. The results triggered a randomized, controlled, double-blind trial in 1449 factory and postal workers. The results of the RCT were quite different. The number of subjects cured at 48 hours in the patulin in buffer arm was 87/668 (13%) vs. 88/680 (13%) in the buffer alone arm, a difference of 0%, 95% CI = (−3.6%, 3.8%), p = 0.96.Citation26
The validity of an ECT depends on the assumption that controls have the same distribution of important baseline characteristics compared to the participants in the test intervention arm. Appropriate analyses include a between-group comparison of these characteristics that can potentially confound the results if imbalanced.
Statistical methods that adjust for the potentially confounding effects of imbalances are often utilized. For example, propensity scores may be used to adjust for differences in patient characteristics based on baseline characteristics that are known and measured. Multivariable regression modeling, stratification, restriction, matching, or instrumental variable methods may also be utilized.Citation27 Most of these adjustments require raw patient-level data which are sometimes unavailable for the controls. Unfortunately using modern statistical methodologies cannot completely address the biases associated with ECTs and invariably depend on untestable assumptions. Some important factors may be beyond our current medical understanding and thus unknown or may not have been measured in the ECT (i.e., known and measured variables are only the tip of the iceberg) resulting in unmeasured confounding and making it impossible to adjust for these factors.
The requirements for a valid ECT are often challenging to demonstrate in the CRE setting. For many CRE infections, patient factors play (e.g.,, age) an important role in explaining observed outcomes. Unfortunately many important variables were not measured in historical studies, making it impossible to control for these factors or confirm a balance of important factors.
In addition, medical practice is constantly changing. In a 10-year longitudinal study conducted at a single ICU,Citation28 the mortality rate decreased despite the rise of resistant bacterial infections. The authors attributed the decrease in mortality to improvements in technology and critical care. Such improvements in standard medical practice confound the results of HCTs for CRE such that observed decreases in mortality could not be attributed to the differences in the interventions being tested.
Resistance patterns are also constantly evolving. Trials conducted in the future will be conducted in diseases caused by CRE with different resistance profile characteristics and different patient characteristics compared to past trials. This again would violate the requirement that controls and test group participants should have the same distributions of important baseline factors.
Studies published in the literature in the CRE setting indicate inconsistent results on objective outcomes such as all-cause mortality, challenging the validity of an HCT in the CRE setting. A meta-analysis of deaths attributable to CRE infectionsCitation29 showed non-uniform failure and substantial variation in mortality outcomes, with point estimates for survival ranging from 6% to 70% across 9 studies. Given the clinical heterogeneity and variation of mortality outcomes in these settings, the interpretation of the results of future HCTs would be very difficult to put into context and interpret.
ECTs are subject to the bias of observational studies. The criteria for a valid ECT should be carefully evaluated before these designs are implemented (). Given the considerable variation in study results in the CRE setting, the lack of information on important patient characteristics that may confound estimates of treatment effects, as well as the improvements in medical practice and evolving antibiotic resistance, the use of ECTs in the CRE setting, should be limited to when RCTs are not possible.
Table 2. Requirements for a Valid Externally-Controlled Trial (adapted from Gehan, 1984 and ICH-E10).Citation24,30
We hope that this study leads to more attention and awareness of the principles of epidemiological study design as applied to clinical investigations of antibiotic resistant bacteria like CRE. We hope that sounder methodology will lead to interventions that can curb the emergence of CRE.
Disclosure of potential conflicts of interest
No potential conflicts of interest were disclosed.
Additional information
Funding
References
- Sievert DM, Ricks P, Edwards JR, Schneider A, Patel J, Srinivasan A, Kallen A, Limbago B, Fridkin S; National Healthcare Safety Network (NHSN) Team and Participating NHSN Facilities. Antimicrobial-resistant pathogens associated with healthcare-associated infections: Summary of data reported to the national healthcare safety network at the centers for disease control and prevention, 2009–2010. Infect Control Hosp Epidemiol 2013; 34(1):1-14; PMID:23221186; https://doi.org/10.1086/668770
- Centers for Disease Control and Prevention (CDC). Biggest threats. http://www.cdc.gov/drugresistance/biggest_threats.html. Accessed 04/14/2015.
- Rothman KJ, Greenland S, Lash TL. Modern epidemiology. 3rd ed. Philadelphia, PA: Lippincott Williams & Wilkins; 2008.
- Dohoo I, Martin W, Stryhn H. Methods in epidemiologic research. Prince Edward Island, Canada: VER, Inc; 2012.
- Harris AD, Samore MH, Lipsitch M, Kaye KS, Perencevich E, Carmeli Y. Control-group selection importance in studies of antimicrobial resistance: Examples applied to Pseudomonas aeruginosa, Enterococci, and Escherichia coli. Clin Infect Dis 2002; 34(12):1558-63; PMID:12032889; https://doi.org/10.1086/340533
- Harris AD, Samore MH, Carmeli Y. Control group selection is an important but neglected issue in studies of antibiotic resistance. Ann Intern Med 2000; 133(2):159; PMID:10896645; https://doi.org/10.7326/0003-4819-133-2-200007180-00018
- Schechner V, Temkin E, Harbarth S, Carmeli Y, Schwaber MJ. Epidemiological interpretation of studies examining the effect of antibiotic usage on resistance. Clin Microbiol Rev 2013; 26(2):289-307. [PMC3623381]; PMID:23554418; https://doi.org/10.1128/CMR.00001-13
- Kaye KS, Harris AD, Samore M, Carmeli Y. The case-case-control study design: Addressing the limitations of risk factor studies for antimicrobial resistance. Infect Control Hosp Epidemiol 2005; 26(4):346-51; PMID:15865269; https://doi.org/10.1086/502550
- Gordis L. Epidemiology. 2nd ed. Philadelphia: W.B. Saunders Company; 2000.
- Hosmer DW, Lemeshow H, May S. Applied survival analysis: Regression modeling of time-to-event data. 2nd ed. Hoboken, NJ: John Wiley & Sons, Inc; 2008.
- Hosmer DW, Lemeshow S, Sturdivant RX. Applied logistic regression. 3rd ed. New York: Wiley; 2013:528.
- McGregor JC, Perencevich EN, Furuno JP, Langenberg P, Flannery K, Zhu J, Fink JC, Bradham DD, Harris AD. Comorbidity risk-adjustment measures were developed and validated for studies of antibiotic-resistant infections. J Clin Epidemiol 2006; 59(12):1266-73; PMID:17098569; https://doi.org/10.1016/j.jclinepi.2006.01.016
- McGregor JC, Kim PW, Perencevich EN, Bradham DD, Furuno JP, Kaye KS, Fink JC, Langenberg P, Roghmann MC, Harris AD. Utility of the chronic disease score and charlson comorbidity index as comorbidity measures for use in epidemiologic studies of antibiotic-resistant organisms. Am J Epidemiol 2005; 161(5):483-93; PMID:15718484; https://doi.org/10.1093/aje/kwi068
- Safdar N, Maki DG. The commonality of risk factors for nosocomial colonization and infection with antimicrobial-resistant Staphylococcus aureus, enterococcus, gram-negative bacilli, Clostridium difficile, and candida. Ann Intern Med. 2002; 136(11):834-44; PMID:12044132; https://doi.org/10.7326/0003-4819-136-11-200206040-00013
- Tsiodras S, Pittet D, Carmeli Y, Eliopoulos G, Boucher H, Harbarth S. Clinical implications of Stenotrophomonas maltophilia resistant to trimethoprim-sulfamethoxazole: A study of 69 patients at 2 university hospitals. Scand J Infect Dis 2000; 32(6):651-6; PMID:11200376; https://doi.org/10.1080/003655400459577
- Marcos M, Inurrieta A, Soriano A, Martínez JA, Almela M, Marco F, Mensa J. Effect of antimicrobial therapy on mortality in 377 episodes of enterobacter spp bacteraemia. J Antimicrob Chemother 2008; 62(2):397-403; PMID:18420813; https://doi.org/10.1093/jac/dkn155
- Harris AD, Karchmer TB, Carmeli Y, Samore MH. Methodological principles of case-control studies that analyzed risk factors for antibiotic resistance: A systematic review. Clin Infect Dis 2001; 32(7):1055-61; PMID:11264034; https://doi.org/10.1086/319600
- Wolkewitz M, Allignol A, Harbarth S, de Angelis G, Schumacher M, Beyersmann J. Time-dependent study entries and exposures in cohort studies can easily be sources of different and avoidable types of bias. J Clin Epidemiol 2012; 65(11):1171-80; PMID:23017635; https://doi.org/10.1016/j.jclinepi.2012.04.008
- Beyersmann J, Kneib T, Schumacher M, Gastmeier P. Nosocomial infection, length of stay, and time-dependent bias. Infect Control Hosp Epidemiol 2009; 30(3):273-6; PMID:19193018; https://doi.org/10.1086/596020
- Schumacher M, Allignol A, Beyersmann J, Binder N, Wolkewitz M. Hospital-acquired infections–appropriate statistical treatment is urgently needed! Int J Epidemiol 2013; 42(5):1502-8; PMID:24038717; https://doi.org/10.1093/ije/dyt111
- De Angelis G, Allignol A, Murthy A, Wolkewitz M, Beyersmann J, Safran E, Schrenzel J, Pittet D, Harbarth S. Multistate modelling to estimate the excess length of stay associated with meticillin-resistant Staphylococcus aureus colonisation and infection in surgical patients. J Hosp Infect 2011; 78(2):86-91; PMID:21481492; https://doi.org/10.1016/j.jhin.2011.02.003
- Schmoor C, Schumacher M, Finke J, Beyersmann J. Competing risks and multistate models. Clin Cancer Res 2013; 19(1):12-21; PMID:23172882; https://doi.org/10.1158/1078-0432.CCR-12-1619
- Greenland S. An introduction to instrumental variables for epidemiologists. Int J Epidemiol 2000; 29(4):722-9; PMID:10922351; https://doi.org/10.1093/ije/29.4.722
- Food and Drug Administration, HHS. International conference on harmonisation; choice of control group and related issues in clinical trials; availability. notice. Fed Regist 2001; 66(93):24390-1; PMID:12356096
- Sacks H, Chalmers TC, Smith H, Jr. Randomized versus historical controls for clinical trials. Am J Med 1982; 72(2):233-40; PMID:7058834; https://doi.org/10.1016/0002-9343(82)90815-4
- Patulin Clinical Trials Committee, Medical Research Council. Clinical trial of patulin in the common cold. 1944. Int J Epidemiol 2004; 33(2):243-6; PMID:15082620; https://doi.org/10.1093/ije/dyh028
- Stafford KA, Boutin M, Evans SR, Harris AD. Difficulties in demonstrating superiority of an antibiotic for multidrug-resistant bacteria in nonrandomized studies. Clin Infect Dis 2014; 59(8):1142-7. [PMC4271096]; PMID:24982037; https://doi.org/10.1093/cid/ciu486
- Rosenberger LH, LaPar DJ, Sawyer RG. Infections caused by multidrug resistant organisms are not associated with overall, all-cause mortality in the surgical intensive care unit: The 20,000 foot view. J Am Coll Surg 2012; 214(5):747-55. [PMC3334453]; PMID:22421258; https://doi.org/10.1016/j.jamcollsurg.2012.01.040
- Falagas ME, Tansarli GS, Karageorgopoulos DE, Vardakas KZ. Deaths attributable to carbapenem-resistant enterobacteriaceae infections. Emerg Infect Dis 2014; 20(7):1170-5. [PMC4073868]; PMID:24959688; https://doi.org/10.3201/eid2007.121004
- Gehan EA. The evaluation of therapies: Historical control studies. Stat Med 1984; 3(4):315-24; PMID:6528131; https://doi.org/10.1002/sim.4780030405