Rethinking the Funding Line at the Swiss National Science Foundation: Bayesian Ranking and Lottery

References

• Adam, D. (2019), “Science Funders Gamble on Grant Lotteries,” Nature, 575, 574–575. DOI: 10.1038/d41586-019-03572-7.
   PubMed Web of Science ®Google Scholar
• Alberts, B., Kirschner, M. W., Tilghman, S., and Varmus, H. (2014), “Rescuing US Biomedical Research from its Systemic Flaws,” Proceedings of the National Academy of Sciences of the United States of America, 111, 5773–5777. DOI: 10.1073/pnas.1404402111..
   PubMed Web of Science ®Google Scholar
• Austrian Research Fund. (2020), “1000 Ideas Programme,” Available at https://www.fwf.ac.at/en/research-funding/fwf-programmes/1000-ideas-programme.
   Google Scholar
• Bates, D., Maechler, M., Bolker, B. M., and Walker, S. C. (2015), “Fitting Linear Mixed-Effects Models Using lme4,” Journal of Statistical Software, 67, 1–48. DOI: 10.18637/jss.v067.i01..
   Web of Science ®Google Scholar
• Berger, J., and Deely, J. (1988), “A Bayesian-Approach to Ranking and Selection of Related Means with Alternatives to Analysis-of-Variance Methodology,” Journal of the American Statistical Association, 83, 364–373. DOI: 10.2307/2288851..
   Web of Science ®Google Scholar
• Bieri, M., Roser, K., Heyard, R., and Egger, M. (2021), “Face-to-face Panel Meetings Versus Remote Evaluation of Fellowship Applications: Simulation Study at the Swiss National Science Foundation,” BMJ Open, 11, e047386. DOI: 10.1136/bmjopen-2020-047386.
   PubMed Web of Science ®Google Scholar
• Bland, J. M., and Altman, D. G. (1999), “Measuring Agreement in Method Comparison Studies,” Statistical Methods in Medical Research, 8, 135–160. DOI: 10.1177/096228029900800204.
   PubMed Web of Science ®Google Scholar
• Bürkner, P.-C. (2018), “Advanced Bayesian Multilevel Modeling with the R Package brms,” R Journal, 10, 395–411. DOI: 10.32614/RJ-2018-017.
   Google Scholar
• Bromham, L., Dinnage, R., and Hua, X. (2016), “Interdisciplinary Research has Consistently Lower Funding Success,” Nature, 534, 684–687. DOI: 10.1038/nature18315.
   PubMed Web of Science ®Google Scholar
• Cao, J., Stokes, S. L., and Zhang, S. (2010), “A Bayesian Approach to Ranking and Rater Evaluation,” Journal of Educational and Behavioral Statistics, 35, 194–214. DOI: 10.3102/1076998609353116..
   Web of Science ®Google Scholar
• Cicchetti, D. (1993), “The Reliability of Peer-Review for Manuscript and Grant Submissions – Its Like Deja-Vu All Over Again – Authors Response,” Behavioral and Brain Sciences, 16, 401–403. DOI: 10.1017/S0140525X0003079X..
   Web of Science ®Google Scholar
• Cole, S., Cole, J. R., and Simon, G. A. (1981), “Chance and Consensus in Peer Review,” Science (New York, N.Y.), 214, 881–886. DOI: 10.1126/science.7302566.
   PubMed Web of Science ®Google Scholar
• DORA. (2019), “DORA – San Francisco Declaration on Research Assessment,” Available at https://sfdora.org/.
   Google Scholar
• Fang, F. C., and Casadevall, A. (2016), “Research Funding: The Case for a Modified Lottery (vol 7, e00422, 2016),” Mbio, 7, e00694–16. DOI: 10.1128/mBio.00694-16.
   PubMed Web of Science ®Google Scholar
• Fang, F. C., Bowen, A., and Casadevall, A. (2016), “NIH Peer Review Percentile Scores are Poorly Predictive of Grant Productivity,” Elife, 5, e13323. DOI: 10.7554/eLife.13323..
   PubMed Web of Science ®Google Scholar
• Fogelholm, M., Leppinen, S., Auvinen, A., Raitanen, J., Nuutinen, A., and Väänänen, K. (2012), “Panel Discussion does not Improve Reliability of Peer Review for Medical Research Grant Proposals,” Journal of Clinical Epidemiology, 65, 47–52. DOI: 10.1016/j.jclinepi.2011.05.001.
   PubMed Web of Science ®Google Scholar
• Gelman, A. (2006), “Prior Distributions for Variance Parameters in Hierarchical Models (comment on article by Browne and Draper),” Bayesian Analysis, 1, 515–534. DOI: 10.1214/06-BA117A..
   Web of Science ®Google Scholar
• Gelman, A., and Rubin, D. B. (1992), “Inference from Iterative Simulation Using Multiple Sequences,” Statistical Science, 7, 457–472. DOI: 10.1214/ss/1177011136..
   Google Scholar
• Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., and Rubin, D. B. (2014), Bayesian Data Analysis (3rd ed.), New York, NY: Chapman and Hall/CRC. DOI: 10.1201/b16018.
   Google Scholar
• Goldstein, H., and Spiegelhalter, D. J. (1996), “League Tables and Their Limitations: Statistical Issues in Comparisons of Institutional Performance,” Journal of the Royal Statistical Society, Series A, 159, 385–443. DOI: 10.2307/2983325..
   Web of Science ®Google Scholar
• Guthrie, S., Ghiga, I., and Wooding, S. (2018), “What do we know about Grant Peer Review in the Health Sciences?” F1000Research, 6, 1335. DOI: 10.12688/f1000research.11917.2.
   Google Scholar
• Guthrie, S., Rodriguez Rincon, D., McInroy, G., Ioppolo, B., and Gunashekar, S. (2019), “Measuring Bias, Burden and Conservatism in Research Funding Processes,” F1000Research, 8, 851. DOI: 10.12688/f1000research.19156.1..
   Google Scholar
• Harman, G. (1998), “The Management of Quality Assurance: A Review of International Practice,” Higher Education Quarterly, 52, 345–364. DOI: 10.1111/1468-2273.00104..
   Google Scholar
• Hedeker, D., Mermelstein, R. J., and Demirtas, H. (2008), “An Application of a Mixed-Effects Location Scale Model for Analysis of Ecological Momentary Assessment (EMA) Data,” Biometrics, 64, 627–634. DOI: 10.1111/j.1541-0420.2007.00924.x.
   PubMed Web of Science ®Google Scholar
• Jayasinghe, U. W., Marsh, H. W., and Bond, N. (2003), “A Multilevel Cross-Classified Modelling Approach to Peer Review of Grant Proposals: The Effects of Assessor and Researcher Attributes on Assessor Ratings,” Journal of the Royal Statistical Society, Series A, 166, 279–300. DOI: 10.1111/1467-985x.00278..
   Web of Science ®Google Scholar
• Johnson, V. E. (2008), “Statistical Analysis of the National Institutes of Health Peer Review System,” Proceedings of the National Academy of Sciences of the United States of America, 105, 11076–11080. DOI: 10.1073/pnas.0804538105.
   PubMed Web of Science ®Google Scholar
• Kaatz, A., Gutierrez, B., and Carnes, M. (2014), “Threats to Objectivity in Peer Review: The Case of Gender,” Trends in Pharmacological Sciences, 35, 371–373. DOI: 10.1016/j.tips.2014.06.005.
   PubMed Web of Science ®Google Scholar
• Kaplan, D., Lacetera, N., and Kaplan, C. (2008), “Sample Size and Precision in NIH Peer Review,” PloS One, 3, e2761. DOI: 10.1371/journal.pone.0002761.
   PubMed Web of Science ®Google Scholar
• Klaus, B., and Alamo, D. D. (2018), “Talent Identification at the Limits of Peer Review: An Analysis of the EMBO Postdoctoral Fellowships Selection Process,” bioRxiv, 481655. DOI: 10.1101/481655..
   Google Scholar
• Laird, N. M., and Louis, T. A. (1989), “Empirical Bayes Ranking Methods,” Journal of Educational Statistics, 14, 29–46. DOI: 10.2307/1164724..
   Google Scholar
• Lingsma, H. F., Steyerberg, E. W., Eijkemans, M. J. C., Dippel, D. W. J., Scholte Op Reimer, W. J. M., Van Houwelingen, H. C., and The Netherlands Stroke Survey Investigators. (2010), “Comparing and Ranking Hospitals based on Outcome: Results from The Netherlands Stroke Survey,” QJM: An International Journal of Medicine, 103, 99–108. DOI: 10.1093/qjmed/hcp169..
   PubMed Web of Science ®Google Scholar
• Liu, M., Choy, V., Clarke, P., Barnett, A., Blakely, T., and Pomeroy, L. (2020), “The Acceptability of Using a Lottery to Allocate Research Funding: A Survey of Applicants,” Research Integrity and Peer Review, 5, 3. DOI: 10.1186/s41073-019-0089-z.
   PubMedGoogle Scholar
• Lucy, L., Freedman, J. E., Becker, L. B., Mehta, N. N., and Liscum, L. (2017), “Peer Review Practices for Evaluating Biomedical Research Grants: A Scientific Statement From the American Heart Association,” Circulation Research, 121, e9–e19. DOI: 10.1161/RES.0000000000000158.
   PubMed Web of Science ®Google Scholar
• Mayo, N. E., Brophy, J., Goldberg, M. S., Klein, M. B., Miller, S., Platt, R. W., and Ritchie, J. (2006), “Peering at Peer Review Revealed High Degree of Chance Associated with Funding of Grant Applications,” Journal of Clinical Epidemiology, 59, 842–848. DOI: 10.1016/j.jclinepi.2005.12.007.
   PubMed Web of Science ®Google Scholar
• Morris, T. P., White, I. R., and Crowther, M. J. (2019), “Using Simulation Studies to Evaluate Statistical Methods,” Statistics in Medicine, 38, 2074–2102. DOI: 10.1002/sim.8086.
   PubMed Web of Science ®Google Scholar
• Plummer, M. (2019), “‘rjags’. Bayesian Graphical Models using MCMC.” Available at https://CRAN.R-project.org/package=rjags.
   Google Scholar
• Salanti, G., Ades, A. E., and Ioannidis, J. P. A. (2011), “Graphical Methods and Numerical Summaries for Presenting Results from Multiple-Treatment Meta-Analysis: An Overview and Tutorial,” Journal of Clinical Epidemiology, 64, 163–171. DOI: 10.1016/j.jclinepi.2010.03.016.
   PubMed Web of Science ®Google Scholar
• Scheiner, S. M., and Bouchie, L. M. (2013), “The Predictive Power of NSF Reviewers and Panels,” Frontiers in Ecology and the Environment, 11, 406–407. DOI: 10.1890/13.WB.017.
   Web of Science ®Google Scholar
• Severin, A., and Egger, M. (2020), “Research on Research Funding: An Imperative for Science and Society,” British Journal of Sports Medicine, 55, 648–649. DOI: 10.1136/bjsports-2020-103340.
   PubMed Web of Science ®Google Scholar
• Severin, A., Martins, J., Heyard, R., Delavy, F., Jorstad, A., and Egger, M. (2020), “Gender and other Potential Biases in Peer Review: Cross-Sectional Analysis of 38 250 External Peer Review Reports,” Bmj Open, 10, e035058. DOI: 10.1136/bmjopen-2019-035058.
   PubMedGoogle Scholar
• Shrout, P. E., and Fleiss, J. L. (1979), “Intraclass Correlations: Uses in Assessing Rater Reliability,” Psychological Bulletin, 86, 420–428. DOI: 10.1037/0033-2909.86.2.420..
   PubMed Web of Science ®Google Scholar
• van der Lee, R., and Ellemers, N. (2015), “Gender Contributes to Personal Research Funding Success in The Netherlands,” Proceedings of the National Academy of Sciences of the United States of America, 112, 12349–12353. DOI: 10.1073/pnas.1510159112.
   PubMed Web of Science ®Google Scholar
• van Houwelingen, H. C., Brand, R., and Louis, T. A. (2009), “Empirical Bayes Methods for Monitoring Health Care Quality,” arXiv:2009.03058 [stat].
   Google Scholar
• Volkswagen Foundation. (2017), “Experiment! – In Search of Bold Research Ideas,” Available at https://www.volkswagenstiftung.de/en/funding/our-funding-portfolio-at-a-glance/experiment.
   Google Scholar