The misdirection of public policy: comparing and combining standardised effect sizes

Abstract

Increased attention on ‘what works’ in education has led to an emphasis on developing policy from evidence based on comparing and combining a particular statistical summary of intervention studies: the standardised effect size. It is assumed that this statistical summary provides an estimate of the educational impact of interventions and combining these through meta-analyses and meta-meta-analyses results in more precise estimates of this impact which can then be ranked. From these, it is claimed, educational policy decisions can be driven. This paper will demonstrate that these assumptions are false: standardised effect size is open to researcher manipulations which violate the assumptions required for legitimately comparing and combining studies in all but the most restricted circumstances. League tables of types of intervention, which governments point to as an evidence base for effective practice may, instead, be hierarchies of openness to research design manipulations. The paper concludes that public policy and resources are in danger of being misdirected.

Keywords:

• Effect size
• experimental design
• meta-analysis

Acknowledgements

I am grateful for the insightful comments on earlier drafts from a number of colleagues, including Nancy Cartwright, Matthew Inglis and Sean McCusker, as well as those from the anonymous reviewers.