Measuring the impact of antimicrobial stewardship programs

ABSTRACT

Antimicrobial Stewardship Programs (ASPs) are being implemented worldwide to optimize antimicrobial therapy, and thereby improve patient safety and quality of care. Additionally, this should counteract resistance development. It is, however, vital that correct and timely diagnostics are performed in parallel, and that an institution runs a well-organized infection prevention program. Currently, there is no clear consensus on which interventions an ASP should comprise. Indeed this depends on the institution, the region, and the patient population that is served. Different interventions will lead to different effects. Therefore, adequate evaluations, both clinically and financially, are crucial. Here, we provide a general overview of, and perspective on different intervention strategies and methods to evaluate these ASP programs, covering before mentioned topics. This should lead to a more consistent approach in evaluating these programs, making it easier to compare different interventions and studies with each other and ultimately improve infection and patient management.

KEYWORDS:

• Antimicrobial stewardship
• antimicrobial resistance
• A-teams
• expert review
• one health

1. Introduction

Antimicrobial resistance is a growing problem. One of the major drivers of this disconcerting development is inadequate use of antimicrobials, both in health-care centers and outpatient settings [1], as well as in livestock [2]. The so-called One Health approach targets resistance development on all the before-mentioned levels. Such a broad approach is considered crucial, in order to effectively minimize the worldwide health-care antimicrobial resistance threat [3]. Part of this approach is improving the usage of antimicrobials in health-care centers and outpatient settings, which in turn helps reducing resistance development [4]. Antimicrobial Stewardship Programs (ASPs) are being hailed as a solution to improve antimicrobial therapies and thus result in a better patient outcome and safety. Different national and international guidelines are available for hospitals, long-term care facilities, and general practitioners [5–7]. There is, however, no clear consensus on the impact of different interventions [8,9]. Effects (clinical and financial) in specific settings or patient populations are difficult to compare and/or evaluate. Some interventions might even be redundant or counterproductive, although in general, published results are often favorable [8,9]. This inconclusiveness necessitates performing scientifically sound (cost-)effectiveness studies on ASP interventions [10]. There are multitudes of methods to evaluate several interventions, but in general, they lack uniformity [10–12]. In this review, we will discuss stewardship in general, its (pre)requisites and the main interventions and their approaches for evaluation, thereby giving a general and up-to-date overview.

2. Importance of a broad stewardship program

The term ‘antimicrobial stewardship’ has been coined roughly 20 years ago [13]. Stewardship programs are now being implemented worldwide and hundreds of articles are published yearly [14]. As it became clear that inadequate antimicrobial use (prophylactic and therapeutic) contributes to resistance development, improving antimicrobial usage became a focus for many health-care institutions, using a subset of different interventions [5,8]. However, it is often overlooked that reducing resistance rates should not be the primary goal. The ultimate goal is to improve clinical outcome and patient safety by providing optimal patient care. Patients should be the main focus and they have important questions related to infectious problems: (1) How can I be protected from a (resistant) infection? (2) Do I have an infection, and if yes, what is causing it? (3) What is the optimal treatment to cure it? Answering these questions requires a broader approach than just an ASP and also entails that certain requirements are met. Besides implementing an ASP, an Infection prevention Stewardship Program should be present to ensure that other patients do not get infected by pertinent (resistant) pathogens, which are often easy transmissible. Furthermore, optimal and timely diagnostics that can adequately and rapidly diagnose the patient’s problems are vital (Diagnostic Stewardship Program). Only if all three aspects are covered – optimal treatment, prevention, and diagnostics (an integrated, Antimicrobial, Infection prevention & Diagnostic [AID] stewardship program) – and all involved stakeholders have the necessary meta-competence (meaning a broad understanding of all relevant above-mentioned aspects), health-care centers can optimally treat infectious patients and tackle the development of antimicrobial resistance [15,16]. Because patient transfers between institutions are also pathogen transfers [17], these three aspects should not only be covered within one local center, but in a (regional) health-care network. This entails close collaboration of all health-care facilities (i.e. hospitals, but also general practices and long-term care facilities) within a clearly defined region [18]. Harmonization of guidelines and practices can be a first start regarding this aspect [19]. In the near future, such an integrated AID approach should lead to a more personalized treatment plan, which is optimally adapted to the specifics of each single patient.

3. Importance of diagnostics

It is thus important that adequate diagnostics are performed on time and provide rapid results to have impact on patient care [20]. Ideally, results, including resistance patterns, should be available before the patient is started on antimicrobial treatment. Three parameters influence the value of diagnostic tests: quality, cost, and time. Overall, the sensitivity and specificity of new commercial and often multiplex-based molecular, point-of-care (POC) assays approach the quality of laboratory developed tests. In this situation, lower costs and/or shorter turnaround times become the main drivers for increased value. From a managerial point of view, we introduced the euro-hour concept, comparable with kilowatt-hour, to easily visualize the impact of both parameters [16]. In this concept, the costs of a test are multiplied by the turnaround time and therefore represent the impact of implementing a POC test. POC tests can only have an impact on antimicrobial therapy and patient management if results are timely available, interpreted, and followed-up by a medical specialist [21,22]. When implemented, it increases the probability of a correct (preliminary) diagnosis – including the reduced need for further diagnostics, streamline antimicrobial treatment sooner if needed (thereby also minimizing the risk for toxicity), and improve infection prevention measurements. In the field of oncology, this so-called theragnostics approach is under continuous development during recent years and it would be a powerful tool in personalized infection management as well [15,16]. Examples of POC tests or rapid diagnostics (e.g. multiplex PCR and matrix-assisted laser desorption/ionization time-of-flight mass spectrometry) already implemented for ASPs are: methicillin-resistant Staphylococcus aureus (MRSA) screening and testing [23–25]; resistance screening [26]; the use in septic/bacteremic patients [27–29]; use of biomarkers (of which procalcitonin probably shows the most promising results) [30–33]; and with viral infections, such as for respiratory illness [19,34].

4. Basics of ASPs

Often ASP interventions are subdivided into three groups: the front-end approach, the back-end approach, and supplemental interventions. Front-end interventions focus on the start of empirical therapy such as pre-analytic consultations and guidelines for (empiric) therapy. Back-end interventions focus on optimization of therapy after, for example, 2 or 3 days. For example, an intravenous (IV)-to-oral switch promotion, de-escalation, or timely stop of therapy, as appropriate. Finally, there are interventions that supplement an ASP such as using resistance data to keep local guidelines up to date and the availability of educational programs [5]. The evaluation of the program is also an important aspect of the latter group. Such a different array of interventions implies that the timeline of impact of an ASP varies broad, with effects on the short-, middle-, and long-term (see Figure 1 for a schematic overview).

Figure 1. Schematic overview of expected timeline of impact of a subset of ASP different aspects.

An ASP should focus on improving patient care and safety, by increasing appropriateness of all antimicrobial use (i.e. prophylactics, empiric therapy, and directed therapy). When looking at prophylactic antimicrobials, there are special cases like Selective Digestive (or Intestinal) Decontamination (SDD) and Selective Oropharyngeal Decontamination (SOD). SDD and SOD are generally implemented at intensive care units (ICUs), and thus are often not part of a standard ASP. They are, however, important forms of antimicrobial use that can influence resistance development and will impact overall policy of antimicrobial usage. It is thus imperative to take these interventions into account when implementing (and evaluating) antimicrobial use and/or intervention strategies [35,36].

5. Unresolved issues with evaluating ASPs

The recent systematic Cochrane review on interventions to improve antimicrobial therapy systematically looked at all studies describing one or more interventions and evaluated their quality and strength of evidence [8]. This review mainly focuses on the clinical effects. One of the main conclusions that can be drawn from the review is the lack of quality of evaluations reported. This is exemplified by the fact that the majority of the studies could not be included [8]. A recent highly comprehensive systematic review and meta-analysis of 14 different antimicrobial stewardship objectives found similar results, albeit generally with a low quality of evidence [9].

Financial effects are equally important. In 2015, two reviews on financial evaluations of ASP studies were published. Both conclude that ASPs are evaluated inconsistently and often even poorly, making it almost impossible to compare studies with another [37,38]. Keeping these results in mind, we will provide a general overview of the different methods to evaluate ASP interventions both clinically and financially (leaving out structural and process-focused aspects). Furthermore, we will mention some pros and cons for each method.

6. Different methods of evaluating ASPs

Randomized controlled trials (RCTs) are considered as the gold standard and most preferable type of study. However, they are often less suitable for antimicrobial intervention studies, due to logistics, ethics, and costs. Nevertheless, in recent years there were a couple of examples looking at ASP interventions in a randomized controlled manner [39,40]. The large majority of published evaluations are however observational studies (e.g. case-control studies, interrupted time series [ITS] analyses, etc.) [8,9,14]. For these studies, comparable cohorts of patients are a major source for bias. This can be even more influenced by changes over time, because the control period is usually several years prior to the intervention period.

With regard to economic evaluations, the preferred method would be to do a cost-effectiveness/-utility study from a societal perspective [41]. Generally speaking, the level of expertise and the time required to do such an analysis are often too scarce to be practically accessible. In practice, this has led to the fact that economic evaluations performed on ASPs are often cost-minimization analyses [37]. Finally, of relevance is the fact that ASPs are implemented on specific wards and/or for specific patient groups (e.g. ICUs, long-term care facilities, septic patients, pediatric patients, and MRSA infections). This makes comparability difficult and it is therefore essential to mention in detail the patient characteristics, as well as the setting of implementation.

7. Clinical outcome measures

The most important goal for an ASP should be to improve quality of patient care. A number of different measures are used that describe some aspects of clinical outcomes. Most important are mortality rates. In general, most studies that evaluate mortality conclude it is not compromised and that an ASP is thus a safe intervention (a non-inferiority analysis). Especially in ASPs targeted at the more severe patient groups (e.g. septic patients), mortality can be an important outcome. For less severe infections (e.g. urinary tract infections [UTI]), the use of mortality as an outcome measure might be less informative. Length of stay (LOS), (secondary) infection rates, and readmission rates are also often measured and evaluated [8,9]. Other less frequently studied outcomes are toxicity and possible side effects (e.g. catheter-related infections or phlebitis) (see Table 1). LOS is one of the more accessible variables to obtain. It is, however, important to take possible secular trends toward earlier discharge into account when evaluating an ASP in a case-cohort setting, especially if the time-period spans multiple years (e.g. did the institution in general saw a drop in LOS over time). Besides the overall hospital LOS, ward-specific LOS (such as ICU stay) is an option. The latter is of course most interesting if the program is also ward-specifically implemented. If treatment improves and infections are cured more effectively, the relapse rates may decrease and readmission rates consequently will go down. However, this outcome measure is biased if there are other hospitals in the vicinity where patients might be readmitted, for example, in clusters of academic centers and surrounding general hospitals. As a more indirect effect, the infection rate for Clostridium difficile can be taken as an outcome measure (see e.g. Nathwani et al., for a successful program [42]). In some studies, a direct correlation of C. difficile infection with antimicrobial use is suspected, especially regarding cephalosporins and clindamycin [43]. This rate might consequently be used as an indirect indicator for antimicrobial use and therefore as an ASP outcome measure.

Table 1. Overview of different outcome measures and some general remarks.

Outcome measures	Remarks
Clinical
Mortatility	Important, but less suitable for mild infections (e.g. uncomplicated UTI)
LOS	General or ward-specific (e.g. ICU stay); easy to obtain, but highly sensitive to biases
Complications	Complications such as IV catheter-related problems and phlebitis
Clostridium difficile	Also indirect measure for antimicrobial use
Readmission rates	Due to relapse, but important to consider effect of neighboring institutions
Toxicity (systemic)	Most frequently in renal function and liver
Other complications	For example, IV catheter-related thrombosis and noninfective phlebitis
Microbiological
Resistance levels	Difficult to measure due to generally long time frame (months to years)
Antimicrobial consumption
Total use	Often measured in DDDs; however, discrepancies are known due to generalization
IV/PO ratio	Of interest with an active IV-to-PO switch program
Broad/narrow ratio	Potentially relevant with regard to resistance development
Appropriateness of therapy	Labor intensive and possibly subjective, but of importance
Financial
Cost–benefit ratio	Preferably done as cost-effectiveness study, including all costs and benefits (at least at hospital level, but preferably at societal level)

UTI: urinary tract infection; ICU: intensive care unit; PO: per os; LOS: Length of stay; DDDs: defined daily doses; IV: intravenous.

8. Microbiological outcome measures

Besides the important clinical outcomes (that directly impact patient care and safety), a secondary important goal is the reduction of resistance levels (see Table 1). There are multiple ways to evaluate this goal. Resistance levels can be measured as percentage of patients (or cultures) with microorganisms ‘resistant’ for a certain antibiotic compared to the number with ‘susceptible’ microorganisms. This parameter can be measured in infected patients or colonized patients. Furthermore, the rate of infections with a resistant microorganism can be taken as a measure (preferably as a percentage of patients infected with the susceptible variant). Difficulties arise due to the longer time that is required before a change in resistance levels can be observed. In addition, reliability of certain trends in the data is difficult to estimate when looking at small numbers. The long time frame also implies that the influence of possible confounders becomes greater. These slow, subtle changes make that antibiograms have been shown to be inconclusive as separate outcome measures and the application of an ITS analysis is therefore a better and preferred method for resistance measures [44]. Furthermore, the baseline level of resistance is also of importance: countries with high resistance levels will most likely see larger effects in a shorter time, provided there is no major influx of resistant microorganisms. A final complicating factor is that ‘resistant’ bacteria reside in the community and in neighboring health-care centers. If an ASP is not implemented regionally, positive results might not be achievable, namely at referral centers. In this setting, a majority of the patients carrying ‘resistant’ bacteria will come from the surrounding health-care region [17,45]. The quality of evidence of ASP effects on resistance is still low [8,46]. To improve studies specifically looking at correlating antimicrobial use and resistance development, a Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) tool was developed (STROBE-AMS) [46]. Concerning SDD/SOD, there is still discussion regarding possible resistance development and it is therefore highly advisable to monitor possible resistance development when implementing such an intervention [47].

9. Antimicrobial consumption outcome measures

ASPs mainly focus on accomplishing changes in broad spectrum IV prescriptions, because broad spectrum antimicrobials are more likely to promote resistance development and IV treatment is more likely to cause secondary infections/complications. The programs focus on either support of the prescribing physician at the start during decision making regarding therapy, or after a few days to support the evaluation of diagnostics and subsequent possible adjustments of therapy. An IV-to-oral switch program is one of the most frequently implemented interventions [48]. To measure the effect on therapy, different outcome measures can be used (see Table 1). Often it is chosen to quantify the antimicrobial therapy as defined daily doses (DDDs, as defined by the WHO), with a denominator correcting for clinical activity such as bed days or admissions (http://www.whocc.no/atc_ddd_index). This can be done either by looking at dispensing data or at purchasing data, which are strongly correlated with each other [49]. From a national point of view, a broader denominator such as inhabitants of the health-care region is also valuable. Because, as mentioned before, people are transferred through different health-care centers within a region, transferring bacteria with them as well. The Infectious Diseases Society of America and the Society for Healthcare Epidemiogy of America advocate DDDs per 1000 patient days as universal outcome measure for ASP programs [5]. This is, however, not always suitable as it is a highly generalizing method that does not take into account patient specifics (such as complicated infections that require high-dose therapy as e.g. endocarditis) and is known to overestimate the use [50]. With respect to pediatric populations, these outcomes are far from optimal, because they are based on adult dosages. This should be taken into account, although up to now it is unclear what measure should preferably be used instead of DDDs [51]. It should thus be noted that a change in DDDs is not entirely suitable for drawing conclusions on the success of an ASP. Optimal antibiotic therapy can also mean that undertreated patients should receive more antibiotics (e.g. deep-seated infections or overweight patients). Personalized therapy measures such as prescribed daily doses (PDDs) or recommended daily doses (RDD) are a more patient-specific approach to quantify antimicrobial treatment and might give more suitable results [50,52]. Furthermore, the length of the therapy (in days) can be evaluated (duration of therapy [DOT]). A discrepancy when compared to DDDs is known due to the difference between administered dose and the WHO DDD values [53]. Because an ASP focuses on optimizing therapy, often by promoting narrow spectrum oral medication, it can be worthwhile evaluating effects of these interventions specifically. This can be done by looking at the percentage of IV medication versus oral (in DDDs/PDDs/RDDs or DOTs) and/or the percentage broad versus narrow spectrum antibiotics (in DDDs/PDDs/RDDs or DOTs). If it is expected that improvements on a ward will be more systemic of nature, it is also worthwhile to evaluate the percentage of patients receiving antimicrobial therapy. Finally, appropriateness of therapy can be evaluated. This is a more labor intensive method, while it often requires reviewing single patient’s files, making it also less objective than ‘hard numbers’ such as DDDs [54]. However, because the goal of an ASP is optimal therapy (according to protocol), appropriateness of therapy is an outcome measure that directly evaluates the main goal of the intervention. An example of this is the analysis with regard to the appropriateness therapy of urinary tract infections [55].

10. Financial outcome measures

With regard to financial evaluations of an ASP, there is much room for improvement [8,37,38]. The most notable issue is that not all costs are taken into account, but just a subset of costs and benefits chosen based on data availability, potentially leading to other cost-effectiveness results. Obviously, for correct interpretation of costs and benefits of a stewardship program, it is important to take into account all costs (and benefits) besides the obvious ones (e.g. antimicrobial costs) [41]. Preferably, this collection of costs should be done prospectively with up front agreed-upon variables and parameters. This minimizes the chances that certain cost types are neglected or cannot be informed. Often various types appear not to be collected when an evaluation is performed retrospectively. Although highly desirable, it is not always necessary or feasible to be cost saving. It is however important to know if the intervention is the most cost-effective way to reach the preferred outcome(s), compared to other potential interventions or the baseline situation. If indeed it is not cost saving, it is worthwhile to take into account a certain threshold of maximum cost per outcome (i.e. cost or willingness to pay per quality-adjusted life year, life-year saved, or other chosen outcome) to enhance optimal allocations of budgets.

An obvious start for integrative costing is to consider all costs that had to be made to implement the program or intervention. This definitely includes time spent by the staff involved, both specifically hired and those already working in the institution. In the latter case, this formally concerns so-called opportunity costs. Furthermore, the required infrastructure (e.g. costs for the introduction of an IT program) and consumables (e.g. extra or new diagnostics) should be considered. In short, all resources and costs of running the program should be included, consistently measured by opportunity costing which reflects the alternative next-best application of these resources and costs (applying to the people involved in the intervention, but also maintenance contracts for IT programs or depreciation costs of laboratory equipment).

If implementation and daily execution costs are known, one can relate these possible savings or benefits and draw conclusions on the cost-efficiency or cost-effectiveness. Preferably, all outcome measures that were evaluated clinically are quantified and transformed into monetary and/or utility values. In general, this will include LOS, antimicrobial use, other procedures done to treat patients (including nursing time), changes in readmissions, infections, and other complications. Quantification into monetary values can be open to interpretation and, for example for LOS high variations in willingness to pay were shown [56], meaning that proper justification is important, inclusive inspection of guidelines for pharmacoeconomic research (http://www.ispor.org).

For ASPs, costs are usually calculated from a hospital perspective regarding monetary outcomes and related to survival and quality-of-life as humanities’ outcomes. When taking a societal perspective, other outcomes should also be included, for example costs due to reduced labor productivity [27]. Unfortunately, ASP evaluations that include financial outcomes often only include direct antimicrobial costs within a very limited perspective, making it nearly impossible to draw conclusions from current literature on full cost-effectiveness of ASPs and hampering comparison with cost-effectiveness of other interventions in health care that are often done from this societal perspective (e.g. drugs) [8,37,38].

11. Expert commentary

ASPs are an important topic with hundreds of publications appearing yearly. In the last few years, asides numerous studies focusing on antibiotics, many studies were published on antifungals with a comparable setup as antibiotic stewardship studies and thus also comparable quality issues [57–59]. Because ASPs are consisting of multiple interventions and not every health-care center is implementing the same interventions, outcome evaluations are also highly diverse. In this respect, a maturity model can for example help to establish the current status of an ASP [60]. This complexity is further increased by the method of evaluation (e.g. RCT, ITS, or case-control study) and different outcome measures used. Finally, there are multiple challenges to obtain high-quality data on effectiveness of an ASP, for example a lack of data of presumptive diagnosis at time of prescribing (or not prescribing) antimicrobials (and subsequent evaluation of appropriateness), as well as exact timing of diagnostics versus start of therapy. Comparing and interpreting different ASP studies is therefore extremely challenging. Cleary, there is a need for appropriate and well-standardized definitions of interventions of an ASP, of the preferred method of evaluation, and of the preferred outcome measures, inclusive those from the financial–economic perspective. Until then, authors should clearly explain and discuss their methods of evaluation in order to make the field of ASPs more transparent.

12. Five-year view

Within the foreseeable future, more tools will be available within daily practice to guide antimicrobial therapy in the best possible way. Faster diagnostics, genomic data, and smarter clinical decision support systems are some of these examples, as well as the growing importance of regional health-care networks and integrative, interdisciplinary collaboration between specialists. This entails that ASPs will continue to develop and that interventions are expected to become easier to implement. Such developments also impact the evaluations of ASPs. It is therefore even more important that ASP evaluations will be performed in a transparent and comparable manner to help streamline the development process.

13. Key issues

• Antimicrobial resistance is continuing to grow and should be considered a global threat for worldwide public health, both from the humanistic and economic perspectives.

• Antimicrobial Stewardship Programs can improve patient care and play an important role in counteracting the threat of resistance.

• It is, however, vital to implement such a program in close collaboration with other healthcare providers from the same healthcare region and with the correct diagnostics and infection control measures in place.

• ASPs consist of many different interventions, depending on local settings. An ASP is therefore often a bundle of interventions, complicating formal evaluation.

• Multiple methods are available for evaluation, inclusive those from the financial-economic perspective.

• Evaluations should be done transparently and with clear description of all relevant aspect to facilitate comparisons between different studies.

Financial & competing interests disclosure

This work was partly supported by the European Union, the German states of North Rhine-Westphalia and Lower Saxony and the Dutch provinces Overijssel, Gelderland and Limburg via the EurSafety Health-net Project [Interreg IVa III-1-01=073]. B Sinha has received a travel grant co-funded by Pfizer/Wyeth, and has worked on projects in cooperation with Pathogenica Life Technologies, and Copan. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.