Evidence‐Based Medicine Can’t Be…

Abstract

Evidence‐based medicine (EBM) puts forward a hierarchy of evidence for informing therapeutic decisions. An unambiguous interpretation of how to apply EBM’s hierarchy has not been provided in the clinical literature. However, as much as an interpretation is provided proponents suggest a categorical interpretation. The categorical interpretation holds that all the results of randomised trials always trump evidence from lower down the hierarchy when it comes to informing therapeutic decisions. Most of the critical replies to EBM react to this interpretation. While proponents of EBM can avoid some of the problems raised by critics by suitably limited the claims made on behalf of the hierarchy, further problems arise. If EBM is to inform therapeutic decisions then a considerably more restricted, and context dependent interpretation of EBM’s hierarchy is needed.

Keywords:

• Evidence‐Based Medicine (EBM)
• Randomised Controlled Trials
• Frequentist Statistics
• Therapeutic Decision Making

Acknowledgements

I would like to thank Mark Colyvan, Jason Grossman, and the participants of the workshop on the concept of evidence in the biohumanities held at The University of Queensland January 2007 for helpful discussion and comments.

Notes

[1] See, for instance, Armitage (1982); Lindley (1982); Suppes (1982); Urbach (1985); Worrall (2007a,b)

[2] A good recent attempt to collate some of the key arguments at the heart of EBM’s claims is provided by Rothwell (2007).

[3] The “methods” referred to by proponents of EBM are certainly not new, and neither is their direct application to the bedside; clinical epidemiology pre‐dates EBM. What proponents of EBM have done, however, is disseminate these ideas, and successfully convince many in the medical community for the application of clinical epidemiological ideas to be the “benchmark” when making, or justifying, medical decisions.

[4] The original evocation of Kuhn is provided in Evidence‐Based Medicine Working Group (1992, 2420); the continued insistence is provided in Guyatt and Rennie (2002, 8).

[5] EBM is most certainly not a paradigm shift in the Kuhnian sense; there is no incommensurability between the new and old theories of medical decision making. Further, the shift to the EBM model of medical decision making has been (and continues to be) piecemeal—this would not be possible if EBM really was a Kuhnian paradigm shift.

[6] Guyatt and Rennie (2002) place N of 1 randomised trials at the top of their hierarchy of evidence. N of 1 trials are conducted with a single patient. In these studies, the patient is randomly allocated to a period of treatment with the intervention under investigation (the “active” treatment) or control. Once the period has ended the patient receives the alternative treatment (either active, or control). The patient’s outcomes are monitored in each period. Both the patient and clinician are blinded to whether the patient is receiving active treatment or control. The set up mimics the very common “unsystematic” clinical practice of giving a patient treatment and monitoring their outcome. N of 1 trials do not play a large role in medical research, and do not assist answering population therapeutic questions. I will not consider them here.

[7] Both Bluhm (2005) and Upshur (2005) recognise the problem of external validity in some form.

[8] Another factor important to external validity, but not related to the basic sciences, are the circumstances under which the trial was performed. If patients included in the trial are treated in ways that are importantly different to how they are treated in routine care, the then external validity of the trial will be low. Assuming the trial treated patients under realistic conditions, then the reliance on the basic sciences to inform judgments about external validity is increased.

[9] In addition, different parts of pathophysiology and pharmacology will have different levels of plausibility. EBM, by placing all of the basic sciences low on the hierarchy, fails to differentiate those parts of the basic sciences in which we have a high degree of confidence with those parts that are currently more speculative in nature.

[10] Worrall (2007a, 983) recognises this point

[11] It might be argued that “evidence‐based” is doing some work in “evidence‐based policy”. Specifically, demarcating policy decisions based on emotion, or tabloid press, from policy decisions based on some form of “evidence”. But, this use of “evidence” is much too vague. To do something more than sound vaguely reassuring “evidence‐based policy” needs to be much more clear about what this “evidence” is, and how it is being used.

[12] See Feinstein (1998); Horwitz et al. (1998), and Rothwell (2005) for discussion. Bluhm (2005) also highlights some of the problems that subgroup analyses hold for EBM.

[13] See Neyman and Pearson (1933) and Neyman (1937).

[14] It should be noted that “power” as defined within hypothesis testing does not play a direct role in estimation theory. However, the conceptual framework for hypothesis testing and estimation are similar, and the influence of a concept similar to power could be outlined within estimation theory. While there are calls within medical statistics, for estimation to completely replace hypotheses testing, p values retain an important role in the analysis of clinical trials Ware et al. (1992).

[15] Although, precisely what should be done about this different epistemic standing is highly contentious. I will, however, leave the details of this debate for another time.

[16] I am also not suggesting that the results of outcomes within trials that have low pre‐trial power are unimportant, or irrelevant (on the contrary these results are very important). My point is simply that the warrant provided for these results according to frequentist statistics is different to the warrant provided for the results of a well‐powered primary hypothesis test. And, that the categorical interpretation of EBM’s hierarchy fails to adequately recognise this difference.