Peer review process: Assessments by applicant-nominated referees are biased, inflated, unreliable and invalid

Abstract

How trustworthy are peer reviews by applicant-nominated assessors (ANAs)? For Australian Research Council (ARC) proposals (N = 2,330) with at least one ANA and one assessor nominated by the funding panel (PNAs), ANAs gave substantially higher ratings in all nine discipline panels (covering sciences, social sciences, and humanities). Compared to reviews by PNAs, ANA ratings were less related to ratings by other assessors, less related to the ARC final assessment, and contributed to the unreliability of peer reviews. Furthermore, when the same assessor was both an ANA and PNA for different proposals, ratings in the role of ANA were biased whereas those by the same person in the role of PNA were not. ANA ratings of ARC grant proposals are biased, inflated, unreliable, and invalid, leading the ARC to abandon use of ANAs. Particularly if replicated in other situations, the results have important implications for other evaluations based on ANAs.

Acknowledgements

This research was funded in part by a grant from the Australian Research Council. We would like to express our thanks to the Australian Research Council and to Professor Max Brennan, former Chair of the Australian Research Council, for assistance in providing the data used in this study. A more detailed description of materials, statistical analyses, and overall research project are available from the Jayasinghe (2004) PhD thesis that is available electronically (http://self.uws.edu.au/Theses/Jayasinghe/list.htm; also see Jayasinghe, Marsh & Bond, 2001; 2003).