Self‐ and Peer‐Assessment Disparities in University Ranking Schemes

Abstract

University rankings often include surveys that involve self‐assessments, but little is known about the effects of these self‐assessments on results. This paper tests for the existence of leniency effects and differential leniency effects, terms borrowed from the performance assessment literature, with a data set of ratings and rankings of Australian university disciplines by Australian academics. It finds support for the leniency effect – these academics rate and rank their own universities higher than others do. It finds less support, however, for the differential leniency effect – academics at the lower‐rated universities do not seem to overrate their universities more than do academics from the higher‐rated universities.

1. The author would like to thank Ross Williams, Eric R.A.N. Smith, Dean Goodman, and Roger Wilkins for their comments on drafts of this article.

Notes

1. The author would like to thank Ross Williams, Eric R.A.N. Smith, Dean Goodman, and Roger Wilkins for their comments on drafts of this article.

2. In the Melbourne Institute rankings, of which I am one of the authors, we survey academics only and include self‐assessments, after determining that the results differ minimally with or without self‐ratings.

3. Although respondents were asked to rank only the top five universities, one can create rankings for the other twenty‐four universities based on average ratings.

4. This ceiling effect also applies to ratings, but is much less pronounced given that the top‐rated university in any discipline received a peer rating of 4.92 on the six‐point scale, which still leaves some room at the top for over‐rating.

5. Calculations may have been differentiated among rankings of 5, 4, 3, 2, and 1, but this still leaves all the rest in a single, ‘unranked’ pool.

6. Available at ⟨www.thes.co.uk/worldrankings⟩. Since this paper was written, THES‐QS has released another league table.

7. QS presentation at the Symposium on “International Trends in University Rankings and Classifications”,Griffith University, 12 February 2007.