And the winner is  …  : inter-rater reliability among scholarship assessors

Abstract

With increasing competition for postgraduate research scholarships, awarding processes demand attention and scrutiny. We examine inter-rater reliability for two prestigious New Zealand scholarships, the Shirtcliffe Fellowship and the Gordon Watson Scholarship. For each scholarship, five assessors (three academic; two non-academic) independently evaluate all applicants over three domains: Academic Merit, Quality of Study Plans and Character/Leadership. Data from years 2009 to 2014 were extracted, comprising 12 separate assessment rounds. Good to excellent agreement was observed for each scholarship in each year. Agreement was significantly higher for the Academic Merit domain compared to the other domains. Moreover, agreement among academics was higher and less variable than non-academics for this Academic Merit domain. No such differences were noted in the other domains. While resource efficiencies could be made, reductions in committee size resulted in poorer applicant selection performance. Applicants and donors alike can be confident that the awardee for these scholarships is a top applicant.

Keywords:

• inter-rater agreement
• scholarship assessment
• reliability
• committee composition and size

Acknowledgements

We thank Mr Jon Winnall and Ms Ciara Zack from Universities New Zealand – Te Pōkai Tara for coordinating the data retrieval and transmission processes. We also wish to acknowledge and thank each of the committee members for consenting to our use of their data, and for their invaluable efforts on these scholarship committees.

Disclosure statement

No potential conflict of interest was reported by the authors.

Supplemental data

Supplemental data for this article can be accessed here http://dx.doi.org/10.1080/03075079.2015.1124849

Notes

1 While the committee has ‘fixed’ memberships, these are time limited. Employing a fixed-effect model would likely have over-inflated consensus estimates. However, given the stability of the committees we considered (13 members for the 10 committee places over the course of the study), our estimates are likely to be conservative.

2 Average CA-ICC is equivalent to Cronbach's alpha (Cronbach 1951) and individual CA-ICC to Pearson's correlation coefficients (McGraw and Wong 1996).