A comparison of two approaches used to gather evidence to support clinical decision making: An editorial

The phrase “evidence-based practice” (EBP) has become common jargon in the health care arena. EBP has been applied so that it affects every aspect of physiotherapy from remuneration to the formats of articles published in peer-reviewed journals. In academic circles, EBP has spawned a new approach to subject reviews and research agendas. Variations on the EBP approach have also become a ubiquitous curricular subject to the point where Massey (Massey, 2003) stated, “Evidence-based practice has been incorporated into all accredited physical therapist education programs”. Recently, Saarni and Gylling (Saarni and Gylling, 2004) recommended that health care professionals pay more attention to different uses of EBP, because the approach to gathering evidence has the potential of being an important factor in political decisions. Academics, administrators and clinicians should understand that their conceptual framework of what information constitutes “evidence” has important ramifications.

EBP has been mistakenly promoted as a generic approach to incorporating scientific findings into top-quality clinical care. However, EBP is one specific approach to judging published literature. From a practical perspective, the types of information and the perceived value of that information can determine subsequent actions. A priori assumptions about the strength of evidence can have an effect on one's judgments. Cultural norms and expectations with regards to information can inhibit or facilitate discussion. The questions that we are allowed to formulate, mechanisms used to gather information, how we value information, or the types of “evidence” that are permitted, will influence our conclusions. Authors have argued that the EBP approach may set up unexpected barriers to clinical problem solving. For example, Sinclair (Sinclair, 2004) argued that the new rituals adopted as part of the EBP process have the potential of limiting thought outside the EBP paradigm.

Popularization of the EBP approach has led to debate (Herbert, Sherrington, Maher, and Mosely, 2001; Powers, 2003) and criticism (Kerridge, Lowe, and Henry, 1998; Sinclair, 2004; Tanenbaum, 1999; Tonelli, 1998; Tygiel, 2001) by several authors. While it is beyond the scope of this editorial to reiterate all of the issues in their entirety, some will be highlighted to contrast two discordant methods of gathering and using evidence. A common shortcoming in previous discussions has been a lack of comparisons with alternative approaches. The phrase “evidence-based practice” has been criticized because it implies that there is only one possible approach to organizing knowledge, and nothing else is “evidenced-based”. For example, Herbert et al. (Herbert, Sherrington, Maher, and Mosely, 2001) state that despite the imperfections of EBP it is the best model of clinical practice. However, the authors do not offer alternative models to support this assertion. Similarly, editorials have voiced concerns about the EBP approach (Baxter, 2003; Tygiel, 2001) but did not offer a systematic, logical and documented approach that could stand as a substitute. Although EBP has gained in popularity recently, and may be the most widely practiced approach to literature review, the approach has strengths and weaknesses that can be compared to other strategies.

There are other formalized approaches to gathering and synthesizing information to support clinical decisions. The traditional literature review approach, as an example, emphasizes comprehensive gathering of available literature and a thorough evaluation of the research. The traditional approach may not meet the needs of physiotherapists because it is often unstructured, verbose and biased. Historically, Mill's Cannons and Henle – Koch's Postulates are two alternative, well-recognized medical approaches to providing evidence for infectious diseases. Sir Austin Bradford Hill (Bradford Hill, 1965) on the other hand, expanded traditional philosophies to identify 9 categories of knowledge that could provide evidence of cause and effect relationships. In a very general sense, many of our clinical decisions are about whether a treatment causes an improvement, or an agent causes a problem. Bradford Hill's approach to cause/effect relationships may have broader application in physiotherapy and may reflect a better manner to understand multifactorial aspects of disease, injury and rehabilitation compared to the EBP format.

The purpose of this editorial is to compare two approaches, Sackett et al. (Sackett, Richardson, Rosenberg, and Haynes, 2000) versus the Sir Austin Bradford Hill approach (Bradford Hill, 1965), to gathering and synthesizing evidence to support a clinical decision (Table 1).

Table 1 Comparison between Evidence Based Practice (EBP) method and Sir Austin Bradford Hill's approach to gathering evidence

EBP	Sir Austin Bradford Hill
A process used to answer a clinical question	An approach to organizing knowledge to show evidence for a cause-effect relationship
5-step linear process	9 categories of knowledge
Research design strategies are critical in accepting information into the decision-making strategy. A hierarchy of research designs.	All research design strategies may be appropriate depending on the circumstances
Expert opinion and consensus statements are largely ignored	Expert opinion and consensus statements are mechanisms to provide a synthesis of knowledge in an ongoing debate
A mechanism by which a treatment has its effect is not required	Must have a good understanding of how an agent produces change
Requires strong evidence to support a clinical decision	Decisions are always made on incomplete evidence. Decisions can be made on weak, moderate or strong evidence depending on the context.
Physiological rational is of limited value	Physiological and foundational science rational are critical to the debate
Questions are answerable	Information is always incomplete
Knowledge/information can be objective, unbiased and accessible	Tests of significance are unimportant Interpretation of data and observations are fallible

SCOPE OF EBP AND BRADFORD HILL'S APPROACHES TO EVIDENCE

The work by Sackett et al. (Sackett, Richardson, Rosenberg, and Haynes, 2000) will be used to present the foundation of the EBP approach since there has been some confusion about the scope and nature of the EBP prescribed method of synthesizing evidence. The proponents of EBP should bear some responsibility for this confusion because of apparent contradictions between their 5-step method and their voiced goals.

The EBP approach distills information based on a hierarchy of research designs. From an EBP perspective, good evidence for a therapeutic intervention must match a very specific design where, in part, researchers randomize their subjects to treatment or control groups. Confusion occurs because proponents of EBP have insisted that research is more than randomized clinical trials (Sackett, Rosenberg, Gray, Haynes, and Richardson, 1996) and that integration of clinical experience is key to the successful application in clinical practice. On the other hand, Sackett et al. (Sackett, Richardson, Rosenberg, and Haynes, 2000) clearly state that nothing but randomized trials should even be read for certain types of clinical questions. Systematic reviews according to EBP practice have a limited repertoire of research designs, and generally the discussion focuses on how well the selected studies match the EBP ideal standard. EBP databases, e.g. the Physiotherapy Evidence Database (PEDro, http://ptwww.cchs.usyd.edu.au/CEBP/index.htm) are limited to randomized controlled trials and systematic reviews in physiotherapy. Interestingly, editorials describing EBP verbalize that a variety of research designs may contribute valuable information, but they contradict themselves by detailed descriptions of the research design hierarchy. For example, Glaros (Glaros, 2003) quotes Ciccone in saying, “Studies from lower levels may be better for your purposes, or they may be better in terms of the quality of the study”. However, Glaros (Glaros, 2003) in the next sentence reinforces the hierarchical approach by saying that “low-quality RCTs are given roughly the same weight in the Sackett hierarchy as well-conducted cohort studies”. In another paper, Maher et al. (Maher, Sherrington, Elkings, Robert, Herbert, and Mosely, 2004) define EBP as “best evidence in combination with clinical expertise and patient values”. However, Maher et al. (Maher, Sherrington, Elkings, Robert, Herbert, and Mosely, 2004) purposefully restrict their paper to randomized trials and systematic reviews of randomized trials without discussion about expertise or patient values. The Panel (Philadelphia Panel, 2001a) developed practice guidelines from randomized controlled trials and other population-based studies using a hierarchical scale. Of the 4,981 on low back pain articles using various research designs identified by the Philadelphia panel, the maximum number of papers reviewed in any category was 8 (Philadelphia Panel, 2001b).

The great majority of EBP reviews fall well short of the recommended 5-step process (Sackett, Richardson, Rosenberg, and Haynes, 2000) by dwelling on the first 3 steps: 1) defining the question; 2) gathering information; 3) and appraising the information according to EBP guidelines. I suggest that the best EBP reviews would be those that could follow through completely with the 5-step process by testing the review's prognostication in the clinical environment in combination with qualitative information from the providers and patients.

The alternative approach described here is documented in a landmark address by Sir Austin Bradford Hill (Bradford Hill, 1965) that is widely available on the Internet. Bradford Hill described 9 categories of knowledge that can be used to deduce cause-effect relationships in health. His approach was originally a template for debates about whether an agent has health implications. His approach can be generalized to consider the agent as an intervention, mechanism of injury, drug or disease process. The sum of the evidence, across the categories, provides the scaffolding from which to build a conclusion. The categories of knowledge, slightly modified from Bradford Hill, are:

• Strength of the association between the cause and effect

• Consistency of the association across situations

• Specificity (i.e. the health consequence of the agent should be specific)

• A predictable time-course where the cause must precede the effect

• A measurable dose-response gradient

• Plausibility/compatibility with current knowledge

• Logical coherence of facts (i.e. the information fits together nicely)

• Experimental evidence

• Analogy with other situations

Using Bradford Hill's elements, one can categorize evidence to determine the depth of knowledge for any clinical question that involves a cause-effect relationship. He stated that no category or piece of evidence is infallible, and that only in its entirety can we hope to use the knowledge to understand the relationship between the agent and its effect. The historic debate that precipitated Bradford Hill's address revolved around whether cigarette smoking caused some forms of cancer. Obviously, this question could not be resolved with randomized trials, and most of the information at the time was gathered from case-control studies.

The goals of the EBP and Bradford Hill's approach are often identical, and even though the two approaches come from the same philosophical roots, even with a cursory glance one can see the approaches are very different. From the outset, the expectations of the approaches are from opposite poles and therefore outcomes might also be expected to be as well.

SCOPE OF LITERATURE USED BY THE TWO APPROACHES

EBP proponents state that only “…2% of clinical (peer-reviewed journal) articles are both valid and of immediate clinical use” (Sackett, Richardson, Rosenberg, and Haynes, 2000)(p 3). Similarly, four physiotherapy journals were found to have published only 11% of articles that matched criteria for applicability and scientific rigor between January 2000 and June 2001 (Miller, McKibbon, and Haynes, 2003). Consequently, EBP proponents recommend going to alternate sources of information (Miller, McKibbon, and Haynes, 2003) presumably because it is more likely to match their conceptual framework. EBP proponents only recommend publications that follow the EBP guidelines so the process of integrating knowledge into the clinic becomes isolated and fraternal.

A consequence of accepting the EBP is to examine a relatively small subset of literature in a specific field. The EBP guidelines supersede journal editors and journal reviewers who rate articles according to the relevance, quality, originality and interest to the readership. The EBP approach may exclude important theories and observations. One illustration of EBP is the series “Evidence in Practice” published in the journal Physical Therapy. A single paper sufficed to produce a clinical decision against a treatment in one installment of the series (Ciccone, 2002). In a subsequent issue, 4 discrepant papers were sufficient to support a treatment (Wetherbee and Pellecchia, 2004). In another series of papers, the Philadelphia Panel in developing clinical practice guidelines choose to review 29 of 5,330 papers on knee pain, and the largest number of papers reviewed in a single category was only 6 (Philadelphia Panel, 2001c). The depth of physical therapy knowledge becomes defined by the narrowness of the hierarchical sifting and filtering process.

Bradford Hill's approach, on the other hand, stresses the value of information from any research design that will build a clear and detailed understanding of the 9 categories of knowledge related to the cause-effect relationship. The resulting review would provide a comprehensive understanding of the relationship. However, if a reviewer were unable to provide convincing evidence in a particular category it is indicative of a gap in knowledge. The quality of research remains important in Bradford Hill's approach because good research is more convincing compared to unsound research. A consequence of using this approach is to integrate foundational sciences, the Nagi model of disablement (Nagi, 1965), and a broader base of literature into logical arguments especially where there do not appear to be randomized controlled trials to support an intervention. For example, using the Bradford Hill approach, a biological mechanism for the intervention would be required in addition to population-based studies.

SEEKING AN ANSWER OR ACTING DESPITE UNCERTAINTY

Sackett et al. (Sackett, Richardson, Rosenberg, and Haynes, 2000) describe the first step in the EBP process as “asking an answer question”. A remarkable assumption about this step is that clinical questions are answerable. Another remarkable assumption of the EBP first step has been that answers to clinical questions can be obtained from written material and then translated to good clinical techniques. Several authors have taken issue with this assumption and criticized the EBP approach because information derived from the literature may not be easily translated to the clinical context. Apart from issues debated in the literature such as generalizability of research finding across environments or across patients (Herbert, Sherrington, Maher, and Mosely, 2001; Tanenbaum, 1999; Tonelli, 1998), applying a priori rules across situations (Sinclair, 2004), or whether EBP is ethically sound (Biller-Andorno, Lenk, and Lietitis, 2004; Kerridge, Lowe, and Henry, 1998; Kirschner, 2001) the type of knowledge derived from written information is different compared to clinical knowledge. Basically, professional skills cannot be obtained entirely from printed material in general or from scientific knowledge specifically (Sinclair, 2004; Tanenbaum, 1999; Tonelli, 1998; Tygiel, 2001).

Bradford Hill stated “all scientific work is incomplete – whether observational or experimental”. He focused on the need to take action despite incomplete information. Risks and benefits must be weighed along with our incomplete evidence and knowledge base: a sentiment that is akin to daily experiences in clinical environments. Bradford Hill went on to say that incomplete or changing scientific literature “does not confer upon us a freedom to ignore the knowledge that we already have, or to postpone the action that it appears to demand.” Along this line, Baxter (Baxter, 2003) suggested that reviews should present clinical utility as an overriding feature compared to “tired formulaic conclusions” typical of EBP reviews.

IS AN EXPERT REQUIRED TO INTERPRET CURRENT KNOWLEDGE?

On the one hand, proponents of EBP emphasize that evidence informs clinical decision-making in physiotherapy but does not dominate (Herbert, Sherrington, Maher, and Mosely, 2001). On the other hand, consensus statements and expert opinion, epitomes of clinical experience, are relegated to level “5” (low) in the EBP hierarchy. According to Tanenbaum (Tanenbaum, 1999), “EBP implies that a clinician's experience contributes little to, and may actually subvert, medical knowledge”. Proponents of the EBP approach assume that information to answer the question is accessible to everyone. Sackett et al. (Sackett, Richardson, Rosenberg, and Haynes, 2000) suggest that anyone can answer clinical questions by following their prescribed process.

A consequence of EBP has been a dramatic shift from the traditional approach that emphasized an expert synthesis and analysis towards an approach that enlists non-experts to produce reviews and evaluate research. For example, the APTA Hooked on Evidence Website (http://www.apta.org/hookedonevidence) has been called a “grassroots” effort to develop a database containing current research evidence. The database provides a significant benefit to physiotherapists endeavoring to survey relevant literature because it provides a synopsis focused on EBP concepts (e.g. blinding and “intention to treat analysis”). However, at the present date, the database doesn't provide an expert interpretation of the various studies such as would be found in a journal club format, nor the opportunity to provide critique of non-population based research. Important questions regarding the translation of the research to the clinical environment are left unanswered. In particular, efficacy of the intervention (whether the intervention works under the best conditions) is difficult to assess given the EBP focus on effectiveness.

Bradford Hill emphasizes the logical construction of a debate that ingrates the 9 categories of knowledge. It appears that his approach requires interpretation and decisions. When describing his 9 criteria he states: “What they can do, with greater or less strength, is to help us to make up our minds on the fundamental question….” Consequently, Bradford Hill appears to support more comprehensive knowledge in a field, as would befit an expert reviewer compared to the EBP approach. Depth and breadth of knowledge in the 9 categories would clearly provide stronger support for a cause and effect relationship between an intervention and the health outcome. Categories, such as the relationship being analogous to other situations, may require some expert interpretation that would elude non-experts.

EFFECTIVENESS OF THE EBP RESEARCH DESIGN HIERARCHY IN ANSWERING CLINICAL QUESTIONS

The randomized controlled trial research design allows for an evaluation of the subject groups' mean differences. This design allows the reader to determine if the average change associated with an intervention was clinically and statistically important. Other research designs, such as bench physiology, animal experiments, and mathematical modeling are degraded using the EBP approach. Unfortunately, it is a very simplistic approach to state that most clinical questions can be resolved with randomized controlled trials.

Physiotherapy interventions are combined to produce a treatment program for each individual patient in a similar manner to combination drug therapies in medicine. Saver and Kalafut (Saver and Kalafut, 2001) identified some marked limitations in the EBP process to evaluate the effect of combined therapies. The authors gave an example of seven classes of drugs that may slow the progression of Alzheimer disease, where it would take 128 clinical trials, enrolling 63,500 patients over about 300 years to investigate combination treatments. Numerous clinical trials would also be required for concurrent intrinsic patient factors (or conditions) that affect one another. Often, the goal of the researcher is to provide an unambiguous conclusion that avoids contamination from comorbitities. On the other hand, as a clinician or administrator you would not want to apply the findings of a study group that has important differences compared to your patient. It is very difficult to transfer the group findings to a patient with complications or side effects. Population-based studies generally do not provide enough information to determine if the clinician can generalize the findings to a specific patient. The EBP approach rejects the use of physiological rational for a treatment, and minimizes the importance of between subject differences. Therefore, the clinician must make a subjective judgment about the relevance of the research to their environment.

Even if there were enough subjects, physiotherapy questions do not receive sufficient funding or have sufficient prerequisite background information to justify mounting a large-scale, population-based study. Using Bradford Hill's model, there are more efficient, effective, clinically relevant strategies for physiotherapists. As examples, longitudinal repeated single-subject designs, crossover designs, dose-response curves, mathematical modeling, qualitative, or correlation studies have advantages for physiotherapy research that have already been demonstrated in the medical literature.

CLOSING THE LOOP ON CLINICAL DECISIONS

The fifth step of the EBP process is to evaluate the effectiveness of a clinical decision within the clinical process (Sackett, Richardson, Rosenberg, and Haynes, 2000). Integrating outcome measures into a clinical process allows feedback; evidence that the process actually works. Unfortunately, the EBP process downplays methods that document clinical effectiveness outside the research design hierarchy. Although the self-evaluation of effectiveness is Step 5 in the EBP process, program evaluation, prospective group designs, case studies and practice reviews are downplayed. To support the temporal nature, the strength of association, the consistency of result, and specificity of result as proposed by Bradford-Hill (Bradford Hill, 1965), what better evidence than your own clinical experience?

CONCLUSIONS

Defining the information that constitutes evidence can have important practical implications. The approach used to synthesizing information into clinical decisions determines the physiotherapy professional scope of knowledge. EBP as presented by its proponents is an uncomplicated, stepwise process that will effectively solve clinical problems. EBP is presented by some as the gold standard for information that will ultimately determine patient treatment. On the other hand, EBP remains controversial when attempting to integrate the system into clinical practice and is just one format of a retrospective literature review. There are very few discussions of the impact, efficacy or effectiveness of the EBP approach compared to alternate approaches. EBP has strayed from other methods of assimilating information by emphasizing a particular hierarchy of research design above all else. Consequently, EBP reviews skim the literature for effectiveness studies and present relatively little information to support a decision.

Bradford Hill's categories of knowledge (Bradford Hill, 1965) have been derived from a long line of clinical reasoning. Many different research designs (theoretical, qualitative and quantitative) substantiate or refute arguments. Using each element illustrates the strengths and shortcomings of our current knowledge about the relationship. Bradford Hill emphasizes methods to take action despite incomplete information. Gathering information that supports the concepts of evidence according to Bradford Hill or using an alternative strategy seems to be more relevant to physiotherapy practice. Future studies should compare the effectiveness of the two approaches in answering various clinical questions. Also, future studies should assess whether the answers to clinical questions may differ significantly with different approaches.