Deriving effectiveness information for decision-making

As of January 2006, a new drug benefit program, Medicare Part D, will be made available and effective for Medicare enrollees. Given the risk factor and morbidity profiles of the Medicare population, there will be an interest in the potential outcomes of treatment alternatives, to guide prescribing decisions.

At the core of such evidence-based research, we expect to see increased activity in research, demonstrations and evaluations designed to improve the quality, effectiveness and efficiency of Medicare, as well as other programs, such as Medicaid and State Children’s Health Insurance Program [101]. In fact, Section 1013 of the Medicare Modernization Act (MMA) includes just this language, which authorizes the Agency for Healthcare Research and Quality (AHRQ) to conduct these evaluations. The essential goals of the Section 1013 mandate are to develop valid evidence about the comparative effectiveness of different treatments and appropriate clinical approaches to difficult health problems, such as might be expected in elderly populations [102]. It is most pressing that physicians, other clinicians and decision-makers have this information available through a readily available network. This can be accomplished through the development of valid scientific evidence and methodologies about the outcomes, comparative clinical effectiveness, safety and appropriateness of healthcare items and services. The ultimate goal is to improve the quality, effectiveness and efficiency of healthcare services.

In order to create such a network, a number of concurrent and complimentary studies are necessary. An ideal platform is a thorough system, which allows the analysis of administrative, survey and clinical databases, as well as secondary databases, to compare health outcomes, including risk and benefit. It would allow the evaluation of the effects of insurance benefit and drug formulary structure on health outcomes, clinical economic and cost studies, and utilization review evaluations including investigations of over-use, under-use, safety and inappropriate prescribing of therapies.

As part of delving into the use of databases, as described above, it is expected that new methodologies and instruments will be developed and implemented. These will necessarily be driven by the research questions, and are expected to revolve around questions to develop new ways of identifying over- and under-utilization of therapies. As we move forward with further analyses, we would need to develop computer algorithms for identifying inappropriate prescribing patterns, and statistical research to advance the scientific use of disease registries, clinical repositories and health databases.

It is also important to have the tools for continuous evaluation, to operate and analyze computerized surveillance and monitoring systems. It is likely that, as a result of the increased need for evidence-based research, device, disease or therapeutic registries will be created and made available for analysis. Most important, especially as related to elderly populations, will be the development of electronic surveillance systems to identify adverse drug reactions and sentinel events, and institute postmarket surveillance of new drugs and devices.

Given limited evidence from clinical trials available on elderly populations, there will be a need to conduct prospective observational and interventional studies, to supplement the aforementioned database approaches. It will be relevant, for example, to study the comparative effectiveness and safety of different therapeutic approaches including use of medical devices; evaluate patient and prescriber decision-making tools; perform accelerated studies comparing the clinical effectiveness of common treatment options in the elderly or to identify optimal medical management for varying levels of disease severity.

Hierarchy of evidence

In view of the new need to gather evidence-based treatment information on elderly populations, it is important to review and learn how traditional study designs can inform decisions. A thorough understanding of their shortcomings is also necessary. Indeed, some research designs are more powerful than others in their ability to provide evidence to aid the decision-making process. The term ‘hierarchy of evidence’, as illustrated in Figure 1, is a spectrum of potential sources, beginning with those most likely to provide the best evidence when evaluating healthcare interventions, which compares effectiveness of a certain treatment regimen to alternatives.

Figure 1 illustrates such a hierarchy. The ranking has an evolutionary order, moving from simple observational methods at the bottom, to increasingly rigorous methodologies. The pyramid shape is used to illustrate the increasing risk of bias inherent in study designs as one goes down the pyramid [1]. The randomized clinical trial is considered to provide the most valid and reliable evidence in evaluating interventions in that it creates a condition in which only one variable is altered, in other words, the process used during the conduct of a randomized clinical trial minimizes the risk of confounding factors influencing the results [2]. The hierarchy implies that when we are looking for evidence on the effectiveness of interventions or treatments, properly conducted systematic reviews of randomized clinical trials, with or without meta-analysis, or properly conducted randomized clinical trials, will provide the most powerful form of evidence. A single rigorous randomized clinical trial may also be sufficient to provide valid information to aid decision-making.

Effectiveness versus efficacy research

While efficacy implies that clinical strategies can achieve their stated goal of improving clinical outcomes when used in optimal circumstances, in clinical decision-making, measurements of effectiveness require demonstration that the new drug does more good than harm compared with the old drug when applied to a target group of patients and provided by a particular group of providers [3]. The very essence of effectiveness requires comparison with alternative methods of caring for patients with particular health states. Consequently, demonstrations of effectiveness must be comparative, measuring the clinical course of a group of subjects treated with and without the new drug, with an alternative.

The randomized double-blind clinical trial is the gold standard methodology in terms of internal validity. The word ‘randomized’ implies that the subjects were not allocated by any individual who might knowingly or unknowingly allocate patients with a better risk profile into one of the two treatment groups. The word ‘controlled’ implies that there is a comparison group. The words ‘double-blind’ imply that neither the patient nor the individual assessing the development of outcomes has knowledge of which treatment group the patient was assigned to, thereby minimizing the threat of biased interpretation. The phrase ‘randomized double-blind controlled trial’ is sometimes shortened to ‘randomized-controlled trial’ (RCT). However, sometimes the attempt to achieve methodological purity can substantially compromise the clinical meaningfulness of the results. This happens when the study period is not long enough for potential cumulative effects of a drug to be detected or when placebo control, rather than an active comparator, is used. In addition, in RCTs, adherence to drugs is more favorable than in naturalistic settings. Furthermore, purity of methods can be compromised when the study sample size is not large enough to generate statistically meaningful results. Therefore, RCTs can only demonstrate efficacy, rather than effectiveness.

While RCT data are rich in clinical detail, they do not reflect real-world experience regarding costs and utilization of healthcare services that is especially meaningful to health policy discussions about such matters as coverage of new pharmaceuticals [4]. While detailed health economic outcomes data are increasingly collected in RCTs, the data are only collected under experimental rather than usual care conditions [5]. Effectiveness research (a term we use in preference to the more confining and difficult health services or outcomes research) evaluates the clinical setting and the healthcare system on which it depends. It uses a variety of healthcare assessment techniques and practical clinical trials to inform clinical practice, quality interventions and health policy decisions [6]. Effectiveness research rectifies many of the problems of RCT data, albeit with its own shortcomings.

Using healthcare benefit claims to demonstrate effectiveness presents one of the major challenges in effectiveness research methodology. Healthcare claims data would seem to be a practical way to evaluate health outcomes in nonexperimental settings and to generalize results to a broader population. Government and private sector healthcare claims are important data sources for health outcomes research because they provide economic information on actual medical practice. For example, over 10 years ago, Mitchell and colleagues reviewed the use of Medicare hospital and physician claims data for outcomes research by ten Patient Outcomes Research Teams (PORTs) supported by the US Agency for Healthcare Policy Research (AHRQ). All of the AHRQ-funded PORTs included components that used insurance claims databases to examine outcomes [7].

However, the tradeoff of effectiveness research is that internal validity may be overly compromised in the effort to ensure generalizability. This is a critical issue that needs to be considered in effectiveness research, namely the balance between external and internal validity. At the present time there is not a standard approach to integrating information aimed at maximizing internal validity, with that aimed at establishing external validity [8].

As we progress with the implementation of Medicare Part D, it is expected that the best systems will emerge, which can support decision-making, through evidence from a combination of clinical trials, databases and registries.

Figure 1. Hierarchy of evidence for questions about the effectiveness of a treatment or an intervention [9].

Systematic review of RCTs with or without meta-analysis RCTs, cohort studies, case–control studies, case-series, case reports, opinion. RCT: Randomized-controlled trial.