From Procrustes to Proteus: trends and practices in the assessment of education research

Abstract

This article is a reflection on an area of particular interest in the current research environment, but which has not yet been explored satisfactorily in the education literature: the evaluation of educational research. The particular focus is on the UK context, but the article is informed by comparative evidence from six countries (gathered through analysis of policy and administrative documents, literature review, informal discussion and written requests for information from key persons). It identifies eight recent trends in the evaluation of education research (from performance‐based funding and institutionalisation of assessment, to the de‐sensitivisation of research assessment) and it explores the benefits and perils of three types of assessment procedures (peer review, bibliometrics and econometrics) as they operate at a micro, meso and macro level. The article argues that current evaluations of educational research (particularly those aimed at supporting funding decisions) tend to operate from an instrumental standpoint that largely ignores the epistemic specificity of the various fields, modes or genres of research, the assumptions about knowledge with which they work, and the cultural and social dimensions of research evaluation as a practice.

Acknowledgements

This article draws on work undertaken with the financial and logistic support of the Oxford University Department of Education and the Oxford Institute of Ageing. It was anticipated by presentations given to the British Educational Research Association conference, Warwick, 2006 (Oancea, 2006) and the European Educational Research Association conference, Geneva, 2006. The insightful comments and questions of the audiences to these events are gratefully acknowledged.

Notes

1. The recent British Academy working group on peer review report, launched in September 2007, also recommended that metrics take a back seat in social sciences and humanities, and that their peer review, with proper training and appropriate consideration of costs, remains the main form of research assessment (British Academy, 2007).

2. See, for example, the 1991 White Paper ‘Higher Education—A New Framework’ (Cm 1541) and the 1992 Further and Higher Education Act.

3. See also the reporting of the performance of the ‘research base’ in the PSA target metrics, covering inputs (including expenditure on research), outputs (including people and publications), outcomes (research recognition, citations, training and research quality), productivity—financial (outputs and outcomes related to inputs) and labour (outputs and outcomes related to other measures), and people (research capacity) (OSI—DTI, 2007).

4. ‘Capability funding’ was created for subjects with emerging research cultures on condition of submission of ‘acceptable’ research strategies: nursing, other studies and professions allied to medicine, social work and art and design.

5. Some argue that education was one of the worst hit subjects post‐RAE 2001. Figures produced by the Association of University Teachers show that education, together with environmental studies and business and management studies, topped the charts in terms of percentage of units to receive no funding after RAE 2001 (60.09%) (AUT, 2003). The figure must be interpreted with caution, though, as it does not take into account the number of submissions and the characteristics of the field and of the infrastructure and staff involved. For an analysis of the results of RAE 2001, see Oancea, 2004.

6. For example, the Witness Seminars approach based on methodology developed by the Institute of Contemporary British History (UK Evaluation Forum, 2006, p. 19).

7. For an examination of different forms of peer review, see also Gibbons and Gheorghiou, 1987.

8. Hackett and Chubin (2003, p. 10) see peer review as the epitome of Kuhn’s ‘essential tension’ between originality and tradition.

9. For example, the more comprehensive term ‘scientometrics’ entered public use only after the establishment of the eponymous journal in 1979.

10. A selective database that relies on criteria such as citation impact to index a limited number of publications of the almost 70,000 relevant periodicals that are currently available worldwide. It was initiated in Philadelphia in the 1960s (1963, for natural sciences; 1975, for humanities; 1979, for social sciences) and indexes over 5000 science journals, 1700 social science journals (plus references from around 3300 journals of cross‐disciplinary relevance) and 1000 (plus 7000 cross‐references) arts and humanities periodicals.

11. A comprehensive bibliographical database of periodicals created in 1932 in the USA and based on criteria such as periodicity and audience rather than quality and citations. The main target audience is composed of libraries and publishing houses rather than research units, and as a consequence the database includes mainly bibliographic and commercial information.

12. See also the debates around proposals for the creation of a European Social Science Citation Index (Gogolin et al., 2003; Botte, 2004) and of a European Citation Index for Humanities (European Social Foundation, http://www.esf.org/).

13. Bill Rammell (2006) responded to the HEPI report by arguing that it oversimplified government’s proposals.

14. In its response to the DfES consultation, the Royal Academy of Engineering also expressed concern that ‘over‐reliance on metrics based on institutions’ income … has the potential to reward expensive research rather than good research’ (2006, p. 5). By contrast, the Wellcome Trust felt that external research income was ‘an effective measure of excellence’ in biomedical research, due to the particularities of the field; however, it cautioned against a resource allocation model that ‘value[s] different funders in different ways’ (Wellcome Trust, 2006, p. 3).

15. Earlier in this article, I commented on the risks entailed by confusing productivity with quality and using indicators of volume as proxies for excellence. Similar risks are connected to definitions of quality in terms of natural sciences‐inspired scientificity or in terms of economic potential, user impact (reduced to observable and attributable improvement in practice) or citation impact.

16. Plutarch Lives. Life of Theseus. Translated by John Dryden; Diodorus Siculus, Library of History (Books III–VIII). Translated by Oldfather; C. H. Loeb Classical Library Volumes 303 and 340. London, William Heinemann, 1935. Book IV.59, 59.5.

17. Lord Rees of Ludlow, Lords Hansard, 30 Mar 2006: Column 950.

18. Baroness Sharp of Guildford, Lords Hansard, 30 Mar 2006: Column 961.

19. Proteus, a sea‐god tending the flocks of Poseidon, had ‘the gift to change and change again in many forms’ (Ovid, Metamorphoses, 2.6.) to escape the necessity of prophesying. He is a symbol of versatility, hybridisation, diversity and resistance to moulding, but also of ambiguity and ever‐elusive truth.