In the spirit of William Georgetti: scrutiny of a prestigious national scholarship selection process

References

• Abate, T. 1995. “What’s the Verdict on Peer Review?” Ethics in Research 1 (1): 1.
   Google Scholar
• Birch, P., J. Batten, and J. Batey. 2015. “The Influence of Student Gender on the Assessment of Undergraduate Student Work.” Assessment & Evaluation in Higher Education. doi:10.1080/02602938.2015.1064857 [ Epub ahead of print].
   Web of Science ®Google Scholar
• Bornmann, L., and H.-D. Daniel. 2005. “Selection of Research Fellowship Recipients by Committee Peer Review. Reliability, Fairness and Predictive Validity of Board of Trustees’ Decisions.” Scientometrics 63 (2): 297–320.10.1007/s11192-005-0214-2
   Web of Science ®Google Scholar
• Bourke, S., A. Holbrook, T. Lovat, and P. Farley. 2004. “Attrition, Completion and Completion times of PhD Candidates.” Paper Presented at the AARE International Educational Research Conference, “Doing the Public Good: Positioning Education Research”, Melbourne, November 28–December 2.
   Google Scholar
• Brouns, M. 2000. “The Gendered Nature of Assessment Procedures in Scientific Research Funding: The Dutch Case.” Higher Education in Europe 25 (2): 193–199.10.1080/713669261
   Google Scholar
• Cole, S. 1992. Making Science: Between Nature and Society. Cambridge, MA: Harvard University Press.
   Google Scholar
• Cyranoski, D., N. Gilbert, H. Ledford, A. Nayar, and M. Yahia. 2011. “The PhD Factory. the World is Producing More PhDs than Ever before. Is It Time to Stop?” Nature 472 (7343): 276–279.
   PubMed Web of Science ®Google Scholar
• Emery, J. A., H. W. Meyers, and D. E. Hunter. 1992. “NIH FIRST Awards: Testing Background Factors for Funding against Peer Review.” Journal of the Society for Research Administration 24 (2): 7–15.
   Google Scholar
• Finn, C. 2002. “The Limits of Peer Review.” Education Week 21 (34): 30–34.
   Google Scholar
• Goodman, S. N., J. Berlin, S. W. Fletcher, and R. H. Fletcher. 1994. “Manuscript Quality before and after Peer Review and Editing at Annals of Internal Medicine.” Annals of Internal Medicine 121 (1): 11–21.10.7326/0003-4819-121-1-199407010-00003
   PubMed Web of Science ®Google Scholar
• Jayasinghe, U. W., H. W. Marsh, and N. Bond. 2003. “A Multilevel Cross-classified Modelling Approach to Peer Review of Grant Proposals: The Effects of Assessor and Researcher Attributes on Assessor Ratings.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 166 (3): 279–300.10.1111/rssa.2003.166.issue-3
   Google Scholar
• Johnston, L., and P. J. Schluter. To appear. “And the Winner is …: Inter-Rater Reliability among Scholarship Assessors.” Studies in Higher Education.
   Google Scholar
• McDowell, A., and N. J. Cox. 2001. Logit Transformation. College Station, TX: StataCorp. http://www.stata.com/support/faqs/statistics/logit-transformation. Updated August 2004; minor revisions April 2015
   Google Scholar
• Mertler, C. A. 2001. “Designing Scoring Rubrics for Your Classroom.” Practical Assessment, Research & Evaluation 7 (25). http://PAREonline.net/getvn.asp?v=7&n=25.
   Google Scholar
• Moskal, B. M., and J. A. Leydens. 2000. “Scoring Rubric Development: Validity and Reliability.” Practical Assessment, Research & Evaluation 7 (10). http://PAREonline.net/getvn.asp?v=7&n=10.
   Google Scholar
• Papke, L. E., and J. Wooldridge. 1996. “Econometric Methods for Fractional Response Variables with an Application to 401(K) Plan Participation Rates.” Journal of Applied Econometrics 11 (6): 619–632.10.1002/(ISSN)1099-1255
   Web of Science ®Google Scholar
• Pierie, J. P. E. N., H. C. Walvoort, and A. J. P. M. Overbeke. 1996. “Readers’ Evaluation of Effect of Peer Review and Editing on Quality of Articles in the Nederlands Tijdschrift Voor Geneeskunde.” The Lancet 348 (9040): 1480–1483.10.1016/S0140-6736(96)05016-7
   PubMed Web of Science ®Google Scholar
• Quade, D., and I. A. Salama. 2006. “Concordance of Complete or Right-censored Rankings Based on Spearman’s Footrule.” Communications in Statistics – Theory and Methods 35 (6): 1059–1069.10.1080/03610920600580091
   Web of Science ®Google Scholar
• Roy, R. 1985. “Funding Science: The Real Defects of Peer Review and an Alternative to It.” Science, Technology, and Human Values 10 (3): 73–81.10.1177/016224398501000309
   Google Scholar
• Stemler, S. E. 2004. “A Comparison of Consensus, Consistency, and Measurement Approaches to Estimating Interrater Reliability.” Practical Assessment, Research, & Evaluation 9 (4). http://PAREonline.net/getvn.asp?v=9&n=4.
   Google Scholar
• Tierney, R., and M. Simon 2004. “What’s Still Wrong with Rubrics: Focusing on the Consistency of Performance Criteria across Scale Levels.” Practical Assessment, Research & Evaluation 9 (2). http://PAREonline.net/getvn.asp?v=9&n=2.
   Google Scholar
• Ward, J. E., and N. Donnelly. 1998. “Is There Gender Bias in Research Fellowships Awarded by the NHMRC?” Medical Journal of Australia 169 (11–12): 623–624.
   PubMed Web of Science ®Google Scholar
• Wennerås, C., and A. Wold. 1997. “Nepotism and Sexism in Peer-review.” Nature 387 (6631): 341–343.10.1038/387341a0
   PubMed Web of Science ®Google Scholar