Rethinking the Funding Line at the Swiss National Science Foundation: Bayesian Ranking and Lottery

Abstract

Funding agencies rely on peer review and expert panels to select the research deserving funding. Peer review has limitations, including bias against risky proposals or interdisciplinary research. The inter-rater reliability between reviewers and panels is low, particularly for proposals near the funding line. Funding agencies are also increasingly acknowledging the role of chance. The Swiss National Science Foundation (SNSF) introduced a lottery for proposals in the middle group of good but not excellent proposals. In this article, we introduce a Bayesian hierarchical model for the evaluation process. To rank the proposals, we estimate their expected ranks (ER), which incorporates both the magnitude and uncertainty of the estimated differences between proposals. A provisional funding line is defined based on ER and budget. The ER and its credible interval are used to identify proposals with similar quality and credible intervals that overlap with the provisional funding line. These proposals are entered into a lottery. We illustrate the approach for two SNSF grant schemes in career and project funding. We argue that the method could reduce bias in the evaluation process. R code, data and other materials for this article are available online.

KEYWORDS:

• Bayesian ranking
• Expected rank
• Funding line
• Grant peer review
• Modified lottery
• Posterior mean

Supplemental Materials

An online fully reproducible supplement is provided which uses an R (ERforResearch) package with the implementation of the above presented methodology (see snsf-data.github.io/ERpaper-online-supplement/). The data used in the case studies can be downloaded from Zenodo: https://doi.org/10.5281/zenodo.4531160.

Acknowledgments

We are grateful to Hans van Houwelingen and Ewout Steyerberg for helpful comments on an earlier version of this manuscript and to the National Research Council of the SNSF for fruitful discussions. We also thank Malgorzata Roos for further feedback on the manuscript as well as on the implementation in R.

Notes

1 see https://www.niaid.nih.gov/grants-contracts/understand-paylines-percentiles.