From Procrustes to Proteus: trends and practices in the assessment of education research

References

• Adam , D. 2002 . Citation analysis: the counting house . Nature , 415 : 726 – 729 .
   PubMed Web of Science ®Google Scholar
• Aristotle . 1915/1975 . “ Ethica Nicomachea ” . In The works of Aristotle , Edited by: Ross , W. D. (vol. 9) , Oxford : Oxford University Press .
   Google Scholar
• AUT . 2003 . The risk to research in higher education in England , London : Association of University Teachers .
   Google Scholar
• Ball , P. 2005 . Index aims for fair ranking of scientists . Nature , 436 : 900
   PubMed Web of Science ®Google Scholar
• Blunkett , D. 2000 . Influence or irrelevance: can social science improve government? Speech made by David Blunkett, Secretary of State for Education and Employment, to a meeting convened by the Economic and Social Research Council on 2 February 2000 . Research Intelligence , 71 : 12 – 21 .
   Google Scholar
• Botte , A. Achievement or performance: observation of productivity of educational research by bibliometric tools. A state‐of‐the‐art‐report . paper presented at the European Conference on Educational Research . September 22–25 . University of Crete .
   Google Scholar
• Braun , T. , Glänzel , W. and Grupp , H. 1995 . The scientometric weight of 50 nations in 27 science areas, 1989–1993: Part I. All fields combined, mathematics, engineering, chemistry and physics . Scientometrics , 33 (3) : 263 – 293 .
   Google Scholar
• British Academy . 2006 . Response to the DfES consultation on the reform of higher education research assessment and funding Available online at: http://www.britac.ac.uk/reports/rae‐2006/response.html (accessed 30 April 2007).
   Google Scholar
• British Academy . 2007 . Peer review: the challenges for the humanities and social sciences , London : British Academy . Report of the working group chaired by A. Weale, September
   Google Scholar
• Buxton , M. , Hanney , S. and Jones , T. 2004 . Estimating the economic value to societies of the impact of health research: a critical review . Bulletin of the World Health Organisation , 82 : 733 – 739 .
   PubMed Web of Science ®Google Scholar
• Capron , H. 1992 . Economic quantitative methods for the evaluation of the impact if R&D programmes , Brussels : EC . EUR 14864 EN, Monitor‐Spear Programme, CEC, SRD DG XII.
   Google Scholar
• Cerroni , A. 2003 . Socio‐cognitive perverse effects in peer review. Reflections and proposals . Journal of Science Communication , 2 (3) Available online at: http://jcom.sissa.it/archive/02/03/F020305/ (accessed 21 March 2005).
   Google Scholar
• Chubin , D. E. 1994 . Grants peer review in theory and practice . Evaluation Review , 18 (1) : 20 – 30 .
   Google Scholar
• Clarke , C. 1998 . Resurrecting educational research to raise standards: statement from the new minister responsible for research . Research Intelligence , 66 : 8 – 9 .
   Google Scholar
• CNER—Comité National D’évaluation de la Recherche . 2003 . Évaluation de la recherche publique dans les établissements publics français [Evaluation of public research in French public establishments] , Paris : France .
   Google Scholar
• Council of the Royal Society . 1995 . Peer review—an assessment of recent developments , London : Royal Society .
   Google Scholar
• Dadds , M. and Kynch , C. 2003 . “ The impact of the RAE 3b rating on educational research in teacher education departments ” . In Research Intelligence no. 84.
   Google Scholar
• Dearing Report . 1997 . Sir Ron Dearing. National committee of inquiry into higher education The Dearing Report, 23 July.
   Google Scholar
• DfES . 2006a . Reform of higher education research assessment and funding. A consultation Consultation document, 13 June 2006. Available online at: http://www.dfes.gov.uk/consultations/downloadableDocs/consultationDoocument%20jcutshall2.doc (accessed 30 April 2007).
   Google Scholar
• DfES . 2006b . DfES consultation on the reform of higher education research assessment and funding: summary of responses Available online at: http://www.dfes.gov.uk/consultations/downloadableDocs/RAE%20response%20summary%20250107.doc (accessed 30 April 2007).
   Google Scholar
• Dogan , M. and Pahre , R. 1990 . Creative marginality. innovation at the intersections of social sciences , Oxford : Westview Press .
   Google Scholar
• Dunne , J. 1993 . Back to the rough ground: ‘Phronesis’ and ‘Techne’ in modern philosophy and in Aristotle , London : University of Notre Dame Press .
   Google Scholar
• ESRC . 2006 . DfES consultation on reform of higher education research assessment and funding. ESRC questionnaire Available online at: http://www.esrcsocietytoday.ac.uk/ESRCInfoCentre/Images/dfes_consultation_questionnaire_tcm6‐16002.doc (accessed 30 April 2007).
   Google Scholar
• Frankel , M. S. and Cave , J. , eds. 1997 . Evaluating science and scientists: an east‐west dialogue on research evaluation in post‐communist Europe , Budapesta : CEU Press .
   Google Scholar
• Frederiksen , L. F. , Hansson , F. and Wenneberg , S. B. 2003 . The Agora and the role of research evaluation . Evaluation , 9 (2) : 149 – 172 .
   Google Scholar
• Gauthier , E. 1998 . Bibliometric analysis of scientific and technological research: a user’s guide to the methodology , Canada : Observatoire des Sciences et des Technologies, Statistics Canada .
   Google Scholar
• Geuna , A. and Martin , B. R. 2003 . University research evaluation and funding: an international comparison . Minerva , 41 (4) : 277 – 304 . (original work published 2001).
   Google Scholar
• Gibbons , M. and Gheorghiou , L. 1987 . Evaluation of research: a selection of current practices , Paris : OCDE .
   Google Scholar
• Glänzel , W. 1996 . A bibliometric approach to social sciences: national research performances in 6 selected social science areas, 1990–1992 . Scientometrics , 35 (3) : 291 – 307 .
   Google Scholar
• Gogolin , I. , Smeyers , P. , Garcia Del Dujo , A. and Rusch‐Feja , D. 2003 . European social science citation index: a chance for promoting European research? Roundtable . European Educational Research Journal , 2 (4) : 547 – 593 .
   Google Scholar
• Goodhart , C. A. E. 1975 . Money, information and uncertainty , London : Macmillan .
   Google Scholar
• Hackett , E. J. and Chubin , D. E. 2003 . “ Peer review for the 21st century: applications to education research, paper for a ” . In National Research Council (SUA) workshop Washington, DC February 25.
   Google Scholar
• Hammersley , M. 1997 . Educational research and teaching: a response to David Hargreaves’ TTA lecture . British Educational Research Journal , 23 : 141 – 162 .
   Google Scholar
• Hansson , F. Best practice in research evaluation? How to evaluate and select new scientific knowledge by introducing the social dimension in the evaluation of research quality . paper for the European Evaluation Society conference . October 10–12 , Seville, Spain.
   Google Scholar
• Hargreaves , D. H. Teaching as a research‐based profession: possibilities and prospects . (Teacher Training Agency Annual Lecture) . London : Teacher Training Agency .
   Google Scholar
• HEFCE/AHRC . 2006 . Use of research metrics in the arts and humanities Report of the Expert Group set up jointly by the Arts and Humanities Research Council and the Higher Education Funding Council for England. Available online at: http://www.hefce.ac.uk/research/assessment/reform/metrics.doc (accessed 7 May 2007).
   Google Scholar
• Hillage , J. , Pearson , R. , Anderson , A. and Tamkin , P. 1998 . Excellence in research on schools , London : DfEE .
   Google Scholar
• HM Treasury . 2006a . Chancellor of the exchequer’s budget statement 22 March 2006. Available online at: http://www.hm‐treasury.gov.uk/budget/budget_06/bud_bud06_speech.cfm (accessed 30 April 2007).
   Google Scholar
• HM Treasury . 2006b . Science and innovation investment framework 2004–2014: next steps March 2006. Available online at: http://www.hm‐treasury.gov.uk/media/D2E/4B/bud06_science_332v1.pdf (accessed 30 April 2007).
   Google Scholar
• House of Commons Science and Technology Committee . 2004 . Research assessment exercise: a re‐assessment , Norwich : TSO . (Eleventh report of session 2003–2004)
   Google Scholar
• Humphrey , C. , Moizer , P. and Owen , D. 1996 . Questioning the value of the research selectivity process in British university accounting . Accounting, Auditing and Accountability Journal , 8 (3) : 144 – 164 .
   Google Scholar
• Johnes , G. 1994 . Research performance measurement: what can international comparisons teach us? . Comparative Education , 30 (3) : 205 – 216 .
   Google Scholar
• Johnstone , D. B. 1998 . The financing and management of higher education: a status report of worldwide reforms , Washington, DC : World Bank .
   Google Scholar
• Kostoff , R. N. 1997 . The handbook of research impact assessment , (7th edn) , Arlington : Office of Naval Research . (DTIC Report No ADA296021)
   Google Scholar
• Kostoff , R. N. 1998 . The use and misuse of citation analysis in research evaluation—comments on theories of citation? . Scientometrics , 43 (1) : 27 – 43 .
   Google Scholar
• Leisyte , L. , de Boer , H. and Enders , J. 2006 . “ England—the prototype of the evaluative state ” . In Reforming university governance. Changing conditions for research in four European countries , Edited by: Kehm , B. M. and Lanzendorf , U. 17 – 52 . Bonn : Lemmens .
   Google Scholar
• Lindsey , D. 1978 . The corrected quality ratio: a composite index of scientific contribution to knowledge . Social Studies of Science , 8 (3) : 349 – 354 .
   Google Scholar
• MacRoberts , M. H. and MacRoberts , B. R. 1989 . Problems of citation analysis: a critical review . Journal of the American Society for Information Science , 40 (5) : 342 – 349 .
   Google Scholar
• Mansfield , E. 1980 . Basic research and productivity increase in manufacturing . American Economic Review , 70 : 863 – 873 .
   Web of Science ®Google Scholar
• Marx , W. , Wanitschek , M. and Schier , H. 1999 . Scientometric on fullerenes and nanotubes . Condensed Matters News , 7 (4) : 3 – 7 .
   Google Scholar
• McIntyre , M. E. November 4 1999 . “ Audit, education and Goodhart’s law or taking rigidity seriously. TV programme ” . In BBC November 4 , Transcript available online at: http://www.atm.damtp.cam.ac.uk/people/mem/papers/LHCE/dilnot‐analysis.html (accessed 25 March 2005).
   Google Scholar
• Moed , H. F. , Burger , W. J. M. , Frankfort , J. G. and van Raan , A. F. J. 1985 . A comparative study of bibliometric past performance analysis and peer judgement . Scientometrics , 8 (3–4) : 149 – 159 .
   Google Scholar
• Narin , F. 1994 . Patent bibliometrics . Scientometrics , 30 (1) : 147 – 155 .
   Google Scholar
• Oancea , A. 2004 . The distribution of educational research expertise in the UK—finding from the analysis of RAE 2001 submissions (I–II) . Research Intelligence , 87 : 88
   Google Scholar
• Oancea , A. 2005 . Criticisms of educational research: key topics and levels of analysis . British Educational Research Journal , 31 (2) : 157 – 183 .
   Web of Science ®Google Scholar
• Oancea , A. Procrustes or Proteus? Towards a philosophical dimension of research assessment . paper to the British Educational Research Association annual conference . September 6–9 , Warwick.
   Google Scholar
• Oancea , A. and Furlong , J. 2007 . Expressions of excellence and the assessment of applied and practice‐based research . Research Papers in Education , 22 (2) : 119 – 137 .
   Google Scholar
• OSI—DTI . 2007 . PSA target metrics for the UK research base , Leeds : Evidence Ltd. . Department of Trade and Industry, Office of Science and Innovation, March 2007
   Google Scholar
• Patsopoulos , N. A. , Analatos , A. A. and Ioannidis , J. P. A. 2005 . Relative citation impact of various study designs in the health sciences . Journal of the American Medical Association , 293 : 2362 – 2366 .
   PubMed Web of Science ®Google Scholar
• Piric , A. and Reeve , N. Evaluation of public investment in R&D–towards a contingency analysis . OECD Conference on Policy Evaluation in Innovation and Technology Report . June 26–27 , Paris. pp. 49 – 64 .
   Google Scholar
• Puxty , A. G. , Sikka , P. and Wilmott , H. C. 1994 . Systems of surveillance and the silencing of UK academic accounting . British Accounting Review , 26 (2) : 137 – 171 .
   Google Scholar
• Rammel , B. Speech to the HEPI conference 21 June on research funding and assessment beyond 2008 . Available online at: http://www.hepi.ac.uk/downloads/BillRammell.doc (accessed 30 April 2006).
   Google Scholar
• Ranson , S. 2003 . Public accountability in the age of neo‐liberal governance . Journal of Education Policy , 18 (5) : 459 – 480 .
   Web of Science ®Google Scholar
• RCUK . 2006a . DfES consultation on the reform of higher education research assessment and funding. Submission from Research Councils UK Available online at: http://www.esrcsocietytoday.ac.uk/ESRCInfoCentre/Images/RCUK_RAE_Consultation_Response_tcm6‐16974.pdf (accessed 30 April 2007).
   Google Scholar
• RCUK . 2006b . Efficiency and effectiveness of peer review project consultation Available online at: http://www.rcuk.ac.uk/cmsweb/downloads/rcuk/documents/prconsultation.pdf (accessed 30 April 2007).
   Google Scholar
• RCUK . 2006c . Report of the research councils UK efficiency and effectiveness of peer review project Available online at: http://www.rcuk.ac.uk/cmsweb/downloads/rcuk/documents/rcukprreport.pdf (accessed 30 April 2007).
   Google Scholar
• Rey‐Rocha , J. , Martin‐Sempere , M. J. , Martinez‐Frias , J. and Lopez‐Vera , F. 2001 . Some misuses of journal impact factor in research evaluation . Cortex , 37 (4) : 595 – 597 .
   Google Scholar
• Roberts , G. 2003 . Review of research assessment Report by Sir Gareth Roberts to the UK funding bodies. May 2003. Available online at: http://www.ra‐review.ac.uk/reports/roberts/roberts_summary.doc (accessed 30 April 2007).
   Google Scholar
• Royal Academy of Engineering . 2006 . Reform of higher education research assessment and funding. Response to the department for education and skills Available online at: http://www.raeng.org.uk/policy/responses/pdf/Higher_Education_Research_Assessment.pdf (accessed 30 April 2006).
   Google Scholar
• Royal Society . 2003 . Supporting basic research in science and engineering: a call for a radical review of university research and funding in the UK , London : Royal Society .
   Google Scholar
• Royal Society . 2006 . Response to the DfES consultation on the reform of higher education research assessment and funding Available online at: http://www.royalsoc.ac.uk/displaypagedoc.asp?id=21472 (accessed 30 April 2007).
   Google Scholar
• Sastry , T. 2006 . A dangerous economy: the wider implications of the proposed reforms to the UK research councils’ peer review system Available online at: http://www.hepi.ac.uk/downloads/MicrosoftWord‐28Adangerouseconomy_proposedUKRCreforms_contents.pdf (accessed 30 April 2007).
   Google Scholar
• Sastry , T. and Bekhradnia , B. 2006 . Using metrics to allocate research funds. A short evaluation of alternatives to the research assessment exercise , Oxford : Higher Education Policy Institute .
   Google Scholar
• Seglen , P. O. 1997 . Why the impact factor of journals should not be used for evaluating research . British Medical Journal , 314 : 498 – 502 .
   PubMed Web of Science ®Google Scholar
• Slowey , M. 1995 . “ Reflections on change: academics in leadership roles ” . In Implementing change from within universities and colleges , Edited by: Slowey , M. London : Kogan Page .
   Google Scholar
• Stake , R. E. Evaluation of testing and criterial thinking in education . paper to the American Psychological Association meeting . August 24 , San Francisco, CA.
   Google Scholar
• St. Clair , R. 2006 . Replacing the RAE: the story so far . Research Intelligence , 97 : 21 – 22 .
   Google Scholar
• St. Clair , R. 2007 . BERA response to RCUK consultation on efficiency and effectiveness of peer reviewing . Research Intelligence , 98 : 12
   Google Scholar
• Thackray , A. 1977 . Reflections on the measurement of science . Newsletter on Science, Technology, & Human Values , 19 : 20 – 29 .
   Google Scholar
• Thompson , K. J. 2004 . “ Interdisciplinarity and complexity: an evolving relationship ” . In E:CO (Special Double Issue) Vol. 6(1–2) , 2 – 10 .
   Google Scholar
• Tooley , J. and Darby , D. 1998 . Educational research: a critique. A survey of published educational research (Report) , London : Office for Standards in Education .
   Google Scholar
• UK Evaluation Forum . 2006 . Medical research: assessing the benefits to society , London : Academy of Medical Sciences, Medical Research Council and Wellcome Trust .
   Google Scholar
• UK Funding Bodies . 2005 . RAE 2008. Guidance to panels Ref RAE 01/2005. Available online at: http://www.rae.ac.uk/pubs/2005/01/rae0105.doc (accessed 30 April 2007).
   Google Scholar
• United States of America Congress . 1937 . The National Cancer Institute Act
   Google Scholar
• Universities UK . 2006 . Universities UK’s response to the DfES consultation on the reform of higher education research assessment and funding Available online at: http://www.universitiesuk.ac.uk/consultations/responses/downloads/ResearchAssessmentAndFundingResponse.pdf (accessed 30 April 2007).
   Google Scholar
• van Raan , A. F. Presentation to the NATO . Evaluation of performance and trends in basic and applied research by advanced bibliometric methods. A science policy instrument for nations with an economy in transition . November 25–28 , Prague. Advanced Workshop on Science Evaluation and its Management
   Google Scholar
• Wellcome Trust . 2006 . Response to the consultation on the reform of higher education research assessment and funding Available online at www.wellcome.ac.uk/assets/WTX033895.pdf (accessed 29 August 2007).
   Google Scholar