Sensitivity of university rankings: implications of stochastic dominance efficiency analysis

ABSTRACT

To create their rankings, university-ranking agencies usually combine multiple performance measures into a composite index. However, both rankings and index scores are sensitive to the weights assigned to performance measures. This paper uses a stochastic dominance efficiency methodology to obtain two extreme, case-weighting vectors using the Academic Ranking of Worldwide Universities (ARWU) and Times Higher Education (THE) data, both of which lead to the highest and lowest index outcomes for the majority of universities. We find that both composite scores and rankings are very sensitive to weight variations, especially for middle- and low-ranked universities.

KEYWORDS:

• University rankings
• higher education
• nonparametric stochastic dominance efficiency
• mixed integer programming

JEL CLASSIFICATIONS:

• C12
• C13
• C14
• C15
• I23

Acknowledgments

Authors would like to thank editor and two anonymous reviewers for their constructive and helpful comments. We also thank Brian Hotson for his comments and suggestions. Mehmet Pinar thanks the Research Investment Fund of Edge Hill University for their financial support. Thanasis Stengos would like to acknowledge financial support from the Social Sciences and Humanities Research Council (SSHRC) of Canada.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1 Hazelkorn (2011, 2014) provides a detailed discussion about policy and institutional changes after the world university rankings.

2 For instance, Cyrenne and Grant (2009) find that different types of Canadian universities follow different methods to raise their reputation ranking.

3 For instance, Saisana, d'Hombres, and Saltelli (2011) use three alternative weighting (i.e. factor analysis derived weights, equal weighting and ‘university-specific weighting’ that maximizes that university's performance relative to all other universities).

4 The most popular use of the pair-wise SD analysis in the literature is to determine which population is better in terms of a given well-being dimension compared to the other (see e.g. Atkinson 1970; Shorrocks 1983; Kakwani 1984; Atkinson 1987; Foster and Shorrocks 1988; Ravallion 1994; Davidson and Duclos 2000; Barrett and Donald 2003; Agliardi, Pinar, and Stengos 2017 among many others). This SD comparisons has moved to a multivariate one by analyzing various welfare dimensions and portfolios (see e.g. Post 2003; Kuosmanen 2004; Linton, Maasoumi, and Whang 2005; Duclos, Sahn, and Younger 2006; Agliardi et al. 2012; Delgado and Escanciano 2013; Pinar, Stengos, and Topaloglou 2013; Agliardi, Pinar, and Stengos 2014; Gonzalo and Olmo 2014; Linton, Post, and Whang 2014; Yalonetzky 2014; Agliardi, Pinar, and Stengos 2015; Pinar 2015; Pinar, Stengos, and Yazgan 2015; Pinar, Stengos, and Topaloglou 2017 among many others).

5 For instance, Bazen and Moyes (2012), and Carayol and Lahatte (2014) use pair-wise SD tests to compare the distribution of publication performance of staff members to rank these institutes. In their application, both the quality and quantity of publications are taken into account and all possible pairs of institutes are compared based on the publication performance.

6 Please refer to https://www.timeshighereducation.co.uk/world-university-rankings/ for the THE World University Ranking data and http://www.shanghairanking.com/ for ARWU rankings data (also known as Shanghai ranking).

7 Scaillet and Topaloglou (2010) derive the limiting behaviour by using the Continuous Mapping Theorem in their Lemma 2.1, similar to that of Lemma 1 of Barrett and Donald (2003).

8 Please refer to the http://www.shanghairanking.com/ for the publicly available data set and detailed definition of each performance criteria used for the rankings.

9 See Billaut, Bouyssou, and Vincke (2010) for a discussion on how normalization procedure of ARWU affects rankings, and that the overall scores cannot be compared over time.

10 It should be noted that in order to use all available variables in our analysis, we left out the institutions that lack data for some of performance indicators. In both rankings, ranking providers reallocate weights across other variables if some institutions have missing information for some indicators. However, in order to keep the pre-determined weights the same, we dropped institutions that have missing information for some criteria.

11 For a detailed description of the THE world university rankings methodology, please refer to https://www.timeshighereducation.com/world-university-rankings-2014-15-methodology.

12 e.g. score differences between three scenarios are less than ten for Harvard University, Stanford University, University of California, Berkeley, ESPCI ParisTech, Toulouse School of Economics, Carnegie Mellon University, City University of New York City College, Technion-Israel Institute of Technology, State University of New York Health Science Center at Brooklyn, Brandeis University, University of Texas Southwestern Medical Center, and Weizmann Institute of Science.

Additional information

Funding

This work was supported by Research Investment Fund of Edge Hill University [1PNARM16] and Social Sciences and Humanities Research Council of Canada [401004].