What is evidence? Reflections on the AMEE symposium, Vienna, August 2011

Abstract

In this article, we present a summary of the discussion from the symposium on ‘what is evidence’, which took place at the AMEE conference in 2011. A panel of five academics and clinicians, plus the chair, considered the nature of evidence, in particular in relation to the ‘evidence’ in the best evidence medical education reviews. Evidence has multiple meanings depending on context and use, and this reflects the complex and often chaotic world in which we work and research.

As health professionals, educators and researchers, we use the word ‘evidence’ frequently in the contexts of evidence-based practice (EBP), evidence-guided education and best evidence medical education (BEME). We have all, no doubt, been challenged in both our professional and personal lives by the question: what is the evidence for that? But perhaps we have not reflected so much on what exactly is meant by ‘evidence’. The symposium at the 2011 AMEE conference was precipitated by discussions at BEME meetings about the nature of evidence gathered and synthesized for BEME systematic reviews. BEME reviewers soon realize that the ubiquitous methodology of EBP in medicine, the randomized double blind controlled trial and subsequent meta-analyses of trial data, is not feasible in education research and evaluation. The BEME collaboration is now looking to improve how evidence from its systematic reviews is implemented in professional education practice to improve student learning (Thistlethwaite & Hammick 2010), and therefore the time is right to consider: what is evidence?

Five eminent scholars were invited to present their thoughts on the question before opening the floor to audience comments. In this article, we focus on the five presentations to offer a taste of the rich discussion. Apart from a consensus between the speakers that randomized controlled trials are not the gold standard of evidence-based education, there was a diversity of content. However, a common theme emerged that it is important for evidence to be applied in practice and translated into action: the generation of evidence through research is not an end in itself. This article is not a word-for-word exposition of the symposium but a synthesis of what was said.

The multiple meanings of evidence

The speakers suggested that the health professional education community has the freedom to decide what 'evidence' is and that the meaning of the word depends on the purpose to which it is to be put. Evidence is not an absolute, a ‘one thing’, a simple definition. It is better described as relative or, as Marilyn Hammick proposed, ‘having agency’. Moreover, a postmodern and, for some, controversial view would be that there is no one meta definition of evidence. Different purposes call for different sources and types of evidence, which may be empirical, theoretical or experiential. Tim Dornan reminded us that a stimulus to the formation of the Cochrane collaboration was the publication of research showing that evidence for the efficacy of drugs exists long before they are routinely prescribed. An argument for the statistical synthesis of efficacy evidence followed logically from that observation. Since then, certain assumptions have become inextricably tied to the term 'evidence', yet they are not necessarily linked to it. A pluralistic approach to evidence calls for choices to be made about the purpose to which evidence is to be put, the type of source material, the key stakeholders who are considered and the relationship between empirical evidence and theory.

Evidence as narrative

Trisha Greenhalgh recounted an experience that highlights the difference in conceptualizing evidence across disciplinary boundaries. During a project exploring the nature of evidence, two main groups formed. One (nicknamed the Bayesians) with members from statistics, economics, cognitive psychology, engineering and medicine set about producing a ‘unifying theory of evidence’ which would apply whatever the subject matter, contextual conditions or specific questions. This group began with the premise that ‘evidence’ was necessarily probabilistic and linked to hierarchies of study designs with systematic reviews and controlled experiments at the top, and focused on improving the science of prediction. The other group (the non-Bayesians) was more heterogeneous, with representatives from philosophy, English, sociology, ancient history, medicine (in the person of Trisha), anthropology and education, while law remained unaligned. The non-Bayesians had no desire to produce a unified theory of evidence – indeed, they began from the position that any such theory would be reductive and unhelpful. Rather, their mission was explicitly pluralist and exploratory, with a narrative approach.

Even in legal cases, and even when much of the evidence consists of numerical or technical measurements of one form or another, the case ultimately rests on the coherence of the narrative that draws it all together. The jury must decide whether the story presented by the prosecution or that presented by the defence is more plausible. Perhaps, therefore as Trisha suggested, all evidence is fundamentally narrative in form.

Or evidence as rhetoric

But then, as Trisha continued, the law professor narrated the famous legal case of ‘Brides in the Bath’. In 1912, George Smith claimed that he had gone shopping to buy tomatoes for his wife's supper, and that when he returned he had found her dead in the bath, presumably from an epileptic fit. In 1913, the same man, who by this time had moved house and remarried, claimed that he had gone shopping to buy tomatoes for his new wife's supper, and that when he returned he had found her dead in the bath. And in 1914, again remarried and again relocated, he claimed for a third time that he had gone shopping to buy tomatoes for his wife's supper, and that when he returned he had found her dead in the bath. While one can evaluate each individual bride's death on the basis of narrative causality, when one assesses the significance of all three deaths, the principles of Bayesian reasoning are applied either consciously or unconsciously.

So, the appellation ‘narrative’ was dropped in favour of ‘rhetoric’, which as Aristotle wrote in the fourth century BC is the art of persuasion, comprising three components: logos (the facts), ethos (the credibility of the speaker) and pathos (the appeal to emotions). In their book The New Rhetoric, Perelman and Olbrachts-Tyteca (1969) acknowledged these three dimensions of rhetoric but added a fourth: an understanding of the audience. The audience for any argument will begin with some shared, taken-for-granted assumptions (or perhaps we may call these values). To a room full of finance directors, the factual information needed is a business plan and financial spreadsheet; a credible speaker is one who is known to manage a very large budget; and the best appeal to emotions will take the form of speculation about some financial catastrophe that could occur if X is not done. But to a room full of climate change activists, the key facts need to relate to carbon footprint; a credible speaker is one who lives in a teepee and grows her own vegetables; and the appeal to emotions will be something to do with the size of the hole in the ozone layer.

The realist approach

Thus, some kinds of evidence relate to what is probably true, as in the Bayesian reasoning of statistics and the experimental sciences, and which we hope is reproducible and generalizable from one setting to another. And other types of evidence relate to what is plausibly true for a unique individual case, as in narrative reasoning and rhetorical argument. The realist approach brings these two philosophical worlds together to produce evidence on what generally works for whom, most of the time and given certain contextual preconditions. This evidentiary middle ground is often based on in-depth case studies and reflexive questioning about why on this occasion a particular input produced a particular outcome but on a different occasion it produced the opposite outcome.

The relation of research to evidence

Huw Davies drew on his published work (Nutley et al. 2007) to demonstrate that studies of knowledge creation in diverse settings have the potential to inform debates about what counts as evidence in health professional education. This wider analysis addresses central questions such as how does research inform evidence? He observed that the findings from research are merely inanimate data, which only acquire meaning and the potential to motivate action – that is, they become construed as knowledge or evidence – when processed by a human actor. Thus, Huw noted that research does not speak for itself: research findings need to be translated, set in context and amalgamated with other sorts of data and knowledge (including tacit knowledge and professional experience). So, while systematic review and syntheses are essential, research must also be interpreted in the context of local systems, cultures and resources. That is, the arena where research is meant to influence is itself a crucial mediator of the meaning that can be ascribed to that research. Such a process of interpretation and integration happens iteratively and dynamically.

The argument made then is that knowledge production from research is a deeply social and contextual process: the process of interpretation and integration happens through dialogue in a sociocultural context, driven by awareness of local problems, challenges and decision needs. The extent to which (research) data gather influential power as ‘evidence’ is a function of this social context and the communication patterns within it. Moreover, evidence is what the powerful say it is: within any sociocultural system, there are power dynamics that confer or attenuate legitimacy. Powerful people (however that power has been accrued) are more able to assert that certain data acquire the persuasive mantle of evidence than those less powerful.

Finally, Huw noted an overemphasis on ‘evidence for decision making’ might lead to a relative neglect of the much broader research base that can inform new ways of thinking. So, research that reshapes conceptual categories and mental models may have more influence in the longer run. Such research may challenge long-held assumptions (prompting unlearning), allow a reconceptualization of the issues that need to be addressed (perhaps even reshaping values in the process) or provide new models and frameworks for understanding these issues. Such fundamental shifts in shared logics may in turn open up whole new ways of engaging with workplace problems. In this sense, notions of ‘evidence’ should go far wider than ‘does it work?’ to help us think more deeply and more critically about the work with which we are engaged, the assumptions that underpin it and the values that drive us.

Evidence in relation to BEME reviews

In her role as appraiser of BEME proposals, Marilyn asks the question: does the review team plan to collect evidence that will enable them to meet their research aim or answer their research questions? A strong link between the research evidence they plan to use in their arguments and the nature of those arguments is fundamental to the process. A simple example of this is that if they plan to show that an education intervention results in improved patient care, then they should be planning to collect evidence from primary studies that have measured the impact of that intervention on patient care.

In systematic review research of the type BEME advocates, there is a line below which we might all agree that something is most definitely not eligible as evidence: for example, one anecdote from one partisan educator, about how their education intervention changed clinical behaviour. For BEME, what can be considered as evidence is determined by what sort of decisions about health professional education we are interested in – by our purpose, and the context of those decisions. Aspects such as what is meant in a particular topic, and for a designated group of learners, by effectiveness: are we interested in effectiveness in terms of knowledge, skills or behaviour change; what do we already know about effectiveness for that particular set of educational circumstances; what do we know about the theoretical basis of the education programme?

The decision may be whether a programme, presently delivered face to face, should be delivered online in the context of this being the institution's first online learning programme involving a large number of students. The evidence sought should be of the what works for whom, in what circumstances variety. In contrast, the decision to implement an educational intervention that has been road-tested for practical delivery and learner acceptability (students liked the new way) but not for improvements in learner knowledge or skills demands a different kind of evidence: evidence that shows the new education delivery is at least as effective in terms of learner knowledge or skills or that it improves knowledge and skills, working more effectively than the traditional way. If there is evidence to support this, we should be asking the why questions, and these require evidence of yet another type, evidence that will enlighten us about the mechanisms that lead to effectiveness.

The quality of evidence

An important debate in health professional education has been whether there is a single arbiter of the quality of evidence, and whether Kirkpatrick's (1967) four-level model can be used in that way. Tim Dornan argued that it is inappropriate to regard Kirkpatrick's levels as a ‘hierarchy’ and proposed that the different distribution of Kirkpatrick's levels in different BEME reviews shows just how diverse education evidence is. What is said to be evidence of the poor quality of education research is, in reality, a reflection of its diversity. BEME reviewers, in fact, use the Kirkpatrick model to organize evidence on outcomes rather than as a quality indicator.

Practical perspectives

Like Marilyn and Tim, Ross Scalese has worked with the BEME Collaboration from its early days, so his comments were not so much about trying to come up with a broad definition of ‘evidence’ from a general viewpoint, but were instead framed very much within the BEME context as reflections on what constitutes best evidence from the medical education perspective. Ross was a member of the topic review group (on medical simulation) who published the first BEME systematic review in 2005 (Issenberg et al. 2005), and since that time, this paper has consistently been among the most cited and most downloaded references from Medical Teacher; a subsequent update of this review (McGaghie et al. 2010) has likewise garnered recognition as the most downloaded Medical Education paper in 2010. Ross mentioned these facts not to ‘toot the horn’ of his own review group, but because the apparent popularity of their work suggests that the medical education community finds something therein to be of value. One of the BEME Collaboration's explicit goals is ‘the production of appropriate systematic reviews of medical education which [not only] reflect the best evidence available … [but also] meet the needs of the user’ (BEME 2011). Ross posited that these reviews were successful in meeting the needs of health professions educators because they were conducted using a practical approach.

One definition of ‘practical’ is useful for further discussion: ‘of or concerned with the actual doing or use of something rather than with theories and ideas’ (OED 2003). This notion of maintaining a practical perspective is applicable then to both the manner in which a systematic review is conducted (‘the actual doing’) and the way the gathered evidence is synthesized and reported (considering the ‘use of’ the results/conclusions). Regarding the former, Tim and Marilyn each commented on the importance of carefully refining the research question and the implications of how the inquiry is framed. In planning and actually carrying out the literature search, data extraction and coding, etc., reviewers must balance systematicity and feasibility, and ask ‘How much more understanding do we gain by uncovering every additional shred of evidence?’ The exponential increase in published reports about medical simulation in the 6 years after their initial study meant that Ross's group might never have completed the updated review if they had performed an absolutely exhaustive literature search; instead, although it was still systematic, they conducted a more selective, ‘critical review’, adopting something of the realist approach described by Trisha and her colleagues.

Regarding the output from scholarly activities, educational researchers must keep the practical needs of end-users in mind. As mentioned earlier, an understanding of the audience is essential: harkening back to the legal metaphor, even if all the DNA, ballistics and other scientific ‘evidence’ is solid, it will be useless for making a case if overly theoretical or incomprehensible to a lay jury. Likewise, especially for clinical teachers who may not have time to keep up with the latest articles in education journals, even the most elegantly designed and well-executed systematic review will not yield useful results if the final written report contains excessive jargon, statistical ‘hocus-pocus' (as in some meta-analyses) or philosophizing. This is not to say that a sound theoretical underpinning to our research and practice is unimportant, nor that we should ‘dumb down’ the results and conclusions from systematic reviews. Rather, the challenge for educational researchers is not simply to uncover the evidence in its various forms, but somehow to distil all its complexity, nuance and contextual richness into something that is more than just consumable, but also digestible and even satisfying to our colleagues in health professions education who are looking to apply the best evidence in their teaching practices.

In conclusion

The phenomenon that is evidence needs to be examined alongside the phenomenon that is decision making in health professional education, and possibly other connected phenomena. Views of what is evidence cannot be formed in isolation from related phenomena. In education, these views are best shaped by an awareness and acknowledgement of the complex and often chaotic world of learning and teaching and its evaluation.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.