Evaluation study design – a pluralist approach to evidence

ABSTRACT

There is significant controversy over what counts as evidence in the evaluation of social interventions. It is increasingly common to use methodological criteria to rank evidence types in a hierarchy, with Randomised Controlled Trials (RCTs) at or near the highest level. Because of numerous challenges to a hierarchical approach, this article offers a Matrix (or typology) of evaluation evidence, which is justified by the following two lines of argument. First, a pluralist approach to evidence is defended. Starting from the principle of methodological aptness, it is argued that different types of research question are best answered by different types of study. This article will address some of the key issues in the debate on RCTs (the ethical principles of duty of care and social utility, the role of random allocation, and threats to internal validity) in respect of which two opposing paradigms provide irreconcilable arguments, namely the position that RCTs are the ‘gold standard’ and the opposing position that RCTs are often if not always inappropriate in social settings. The second line of argument is that evaluations often require both experimental and non-experimental research in tandem. In a pluralist approach, non-causal evidence is seen as a vital component in order to evaluate interventions in mixed methods studies (as part of evidence-based practice [EBP]) and also is important for good practice itself (as part of practice-based evidence [PBE]). The article concludes by providing a detailed description of what an Evaluation Evidence Matrix can and cannot do.

KEYWORDS:

• evaluation
• evidence
• hierarchy
• matrix
• pluralist

Notes

1 A within-groups quasi-experimental study collects data from one group, usually before and after exposure to the programme being evaluated. A between-groups quasi-experimental study collects data from two groups, only one of which received the programme, but the treatments are assigned to participants by a non-random process.