On promoting rigour in educational research: the example of the RAE

Abstract

This article offers a deconstruction of the RAE Education sub‐panel’s rubrics, drawing also on the broader RAE regulations, procedures, and associated documentation and research. It seeks to tease out the sorts of covert epistemologising that may (or may not) be likely to take place. The theoretical ambition is to take a Derridean approach to acts of ‘translation’, reworking that metaphor as a way into the acts of translation that the RAE undertakes in placing numerical values on the quality of research outputs. How do we measure our ‘pounds of flesh’? As a practical ‘hermeneutics of suspicion’ it seeks a ‘provocative validity’ in that its aim is to provoke clarity and reassurance from the authors of such RAE documents. In that ambition it has had some success, as the response from Margaret Brown, Chair of the Education sub‐panel, demonstrates.

Acknowledgements

Thanks for critical comment to Jo Frankham and Harry Torrance. An earlier version of this paper was delivered at the British Educational Research Association conference, Warwick University, September 2006.

Notes

1. It may seem strange that a simple numerical scale should become so contaminated with letters and stars, but it’s important also to read these scales as class markers, in the way that socio‐economic classes are distinguished in the UK. To joke only a little, 3b/3a might be seen as corresponding to a distinction between ‘rough’ and ‘respectable’ working class. The economic disenfranchisement of 1/2/3b/3a in 2001 thereby enhanced the loading of funding at the top end and widened the gap between ‘have’ and ‘have‐not’ in ways that might also be regarded as analogous, particularly as these cuts were conducted under the overall policy banner of ‘capacity building’. There are underlying parallels, it might be argued, with New Labour inclusionary rhetorics.

2. There is a longstanding history of ‘significance’ and ‘originality’ in relation to questions of ‘recognition’ in Science and its Sociology (Merton, 1957). The priority of ‘originality’ is clear.

3. Armstrong and Goodyear argue: ‘It would not do for an assessment model to be dominated by advocates of large‐scale, randomised control group experiments (or poststructuralist policy critique, for that matter)’ (2005, p. 21). Indeed. (See also Stronach (2005).)

4. A natural riposte to such questions is to dismiss such possibilities as a slur on the integrity of reviewers. But our recent and highly analogous experience with an ESRC end‐of‐report review shows that such paradigm warfare does happen. The report and case study was reviewed. One reviewer said good things about the research. First, ‘a qualitative approach is justified’. But ‘with some reluctance I have rated this report “problematic” rather than “good”’. Why? The reviewer’s ‘fairly strong reservations derive from my prejudices against qualitative research’. The other three reviewers rated the research ‘outstanding’, which was its overall grade. The example illustrates both the possibilities of bias and its correction. It highlights the need for careful moderation across reviewers and the need to take paradigmatic bias into account.

5. Or perhaps we’re back to ‘rigor’ as in ‘rigor mortis’. MacLure has criticised the fruits of such systematic reviews as illustrated in the Hypothetical Example as ‘tiny dead bodies of knowledge’ (MacLure, 2005).

6. Benjamin differentiates between the ‘intended object’ and the ‘mode of intention’ in making this point in relation to the meaning of ‘bread’ and ‘wine’ in different languages (Benjamin,1973, p. 74). Others might express a similar point in relation to connotative and denotative meanings. The RAE’s move from overall institutional assessment to individual item appraisal has its parallels in the practices of risk management: ‘The so called asset‐by‐asset approach dominates the scene over the portfolio‐theoretical approach’ (Kalthoff, 2005, p. 75).

7. In discussion at the BERA presentation of an earlier version of this paper, I argued that the RAE was like the ‘pool’s panel’—which meets in the UK to decide the result of postponed football matches so that gambling results are available. But that was the 2001 RAE. There is a difference in the 2008 RAE exercise. This time the panel’s wisdom is far greater: no longer content to decide the outcome of a match that was never played, it also decides how well each player performed.