Are peer reviews of grant proposals reliable? An analysis of Economic and Social Research Council (ESRC) funding applications

ABSTRACT

Peer review is widely used throughout academia, most notably in the publication of journal articles and the allocation of research grants. Yet peer review has been subject to much criticism, including being slow, unreliable, subjective and potentially prone to bias. This paper contributes to this literature by investigating the consistency of peer reviews and the impact they have upon a high-stakes outcome (whether a research grant is funded). Analysing data from 4,000 social science grant proposals and 15,000 reviews, this paper illustrates how the peer review scores assigned by different reviewers have only low levels of consistency (a correlation between reviewer scores of only 0.2). Reviews provided by “nominated reviewers” (i.e. reviewers selected by the grant applicant) appear to be overly generous and do not correlate with the evaluations provided by independent reviewers. Yet a positive review from a nominated reviewer is strongly linked to whether a grant is awarded. Finally, a single negative peer review is shown to reduce the chances of a proposal being funding from around 55% to around 25% (even when it has otherwise been rated highly).

KEYWORDS:

• Peer review
• Consistency
• Grant funding

Highlights

• Peer review scores assigned by different reviewers have only low levels of consistency (a correlation between reviewer scores of only 0.2).

• Reviews provided by “nominated reviewers” (i.e. reviewers selected by the grant applicant) appear to be overly generous and do not correlate with the evaluations provided by independent reviewers. Yet a positive review from a nominated reviewer is strongly linked to whether a grant is awarded.

• A single negative peer review is shown to reduce the chances of a proposal being funding from around 55% to around 25% (even when it has otherwise been rated highly).

Acknowledgements

I am grateful to the ESRC, one of the seven Research Councils of UKRI, for sharing their data. This project would not have been possible without their support.

Notes

¹ This statement describes how applicants will “act to enable the research to connect with others and make a difference conceptually and instrumentally.” https://esrc.ukri.org/research/impact-toolkit/developing-pathways-to-impact/?_ga=2.152184825.1305920688.1553508319-271472340.1553508319.

² See nominated reviewer section of https://je-s.rcuk.ac.uk/handbook/index.htm.

³ The database used in this paper suggests that receiving less than three reviews is rare.

⁴ For further details about this six-point scale, see page 3 of https://esrc.ukri.org/files/funding/guidance-for-peer-reviewers/faqs-for-peer-review-college-members/.

⁵ The panel membership as of September 2018 can be found at https://esrc.ukri.org/files/about-us/governance-and-structure/membership-of-the-grant-assessment-panels/.

⁶ See https://je-s.rcuk.ac.uk/handbook/pages/IntroducerAssessment/ESRC_Introducer_Assessment_Guidance.htm.

⁷ The author initially requested the data under Freedom of Information legislation. Although this was rejected, it started a conversation with the ESRC. It was agreed that a limited amount of data could be provided for the purposes of writing this academic paper, with it being kept within a secure server at UCL and not to be further shared.

⁸ Any grant application that involved the author (either as an applicant or as a reviewer) was also excluded from the database that the ESRC provided. Information for the 2018/19 financial year was partial as the data was received part way through this period.

⁹ Note that polychoric (rather than Pearson) correlation is used to account for the categorical nature of ESRC peer review scores. This is a technique for estimating the correlation between two latent variables that are assumed to be continuous and normally distributed, based upon observed ordinal data.

¹⁰ Weighted Kappa statistics give more weight to larger disagreements between reviews (cells are further away from the leading diagonal on the cross-tabulation). Hence a difference between two reviewers who score a proposal 5 and 2 is treated as lower agreement than two reviewers who score a proposal a 4 and 3. (Unweighted Kappa would treat these two situations equally).

¹¹ We conducted robustness tests in which we (a) analyse only proposals with three reviews and (b) analyse proposals with between three and six reviews. These did not substantively alter our findings.

¹² These figures increase marginally to 0.48 (four reviewers) and 0.53 (five reviewers) when nominated reviewers are excluded.

¹³ As noted in the methodology section, these and subsequent figures in this section are computed only for proposals with three or four reviewers.