The suspect statistics of best practices: a triple critique of knowledge production and mobilisation in the global education policy field

ABSTRACT

The present article addresses the production and global dissemination of ‘policy relevant knowledge’. It not only unpacks the methodological assumptions of a particular type of knowledge production – known as impact evaluations – but also analyses the issue of knowledge mobilisation within the political economy of the global education policy field. Having a critical understanding of impact evaluations is crucial because they are widely regarded as the most valid informational basis from which to make policy decisions. The importance of grasping the methodological limitations and political-economic dynamics that afflict knowledge production and mobilisation is demonstrated through the case of Colombia’s well-known charter school programme. By employing a strategy that has been labelled bibliographic ethnography, this article not only takes a critical look at the knowledge base that has been produced on this programme but also maps the way that evaluations of this charter school programme, despite their limitations, have been cited and invoked in academic and organisational publications to project this programme internationally. The article concludes by offering a theoretically-informed discussion of how we should understand the trajectory of impact evaluations (and other knowledge products) as they cross multiple personal, organisational, political, and discursive contexts.

KEYWORDS:

• Impact evaluations
• global education policy
• policy relevant knowledge
• knowledge mobilisation
• knowledge production
• Colombia
• concession
• bibliographic ethnography

Disclosure statement

No potential conflict of interest was reported by the authors.

ORCID

D. Brent Edwards Jr. http://orcid.org/0000-0003-3955-9525

Jeaná Morrison http://orcid.org/0000-0001-9622-3142

Notes

1 Of course, there is a separate literature that presents a critical analysis of the historical and epistemological foundations of those methods (e.g. Gigerenzer et al. 1989; Grosfoguel 2013; Kaomea 2015; Shahjahan 2011; Smith 1999). This literature, however, goes beyond the scope of the current paper.

2 This section offers arguments that have been condensed and revised from chapter two of a recent book by the author on impact evaluations and the political economy of knowledge production (Edwards 2018).

3 Treatment groups contain participants that partake in the intervention of interest while control groups serve as the comparison to understand what happens to those who do not participate in the intervention.

4 The other conditions that we do not address and which are arguably more abstract are (a) the need to ensure that all variables are measured properly and (b) the need to correctly model the interrelationships among the variables included (Klees, 2016; Leamer, 1983). There is continual debate among researchers for how to deal with these two issues, as there is no consensus for how to measure, for example, things like learning or achievement (though in practice researchers tend to use the tests that are available or else develop their own), let alone for how to mathematically model the complex real-world relationships between the variables for which data is available.

5 To be sure, accounting for all possible causes of social phenomena is a challenge faced by all methodologies. The point here is that a pretence of regression analysis is that all variables are included if we are to get accurate estimates of regression coefficients.

6 Some may dispute the characterisation of this study as an impact evaluation because of its simple nature. However, it is included here as such because it sought to say something about impact and because it employed regression analysis, even if in its most basic form.

7 In all, we identified 144 different source texts; one could not be obtained.

8 It is also the case that these were among the earlier publications about the CEC programme which has afforded more time for them to be cited.

9 See Edwards (forthcoming) for a further discussion of how quantitative methods contribute to ‘epistemicide’.