No proxy for quality: why journal rankings in political science are problematic for political theory research

ABSTRACT

Journal rankings for political science have been regularly published, from the 1970s onwards, by the American Political Science Association’s ‘state of the discipline’ journal. Politics journals have also been officially ranked by the Australian Political Studies Association into four bands (A*, A, B and C) from 2007 onwards. This article shows, first, that the assumption grounding these exercises (namely, that disciplinary journal rankings can serve as proxies for the quality of articles in their pages) is undermined by the findings of the broader research evaluation literature, especially with respect to sub-disciplines (like political theory, Australian politics, and some types of qualitative comparative politics) that bear certain characteristics. Next, outlining the findings of a 2018 survey, it is argued that the disciplinary use of journal rankings in political studies not only has damaging effects on research in political theory, but also advantages other sub-disciplines. The paper closes with two recommendations.

1970年代以来美国政治学学会的⟪学科现状⟫杂志会定期公布政治学刊物的排行。2007年起，澳大利亚政治学研究会也将政治学刊物正式分为A*、A、B、C四个行列。根据本文的研究，这类事情所基于的前提（即排行代表了杂志文章的质量），受到更广泛的研究评估、尤其是子学科（如政治理论、澳大利亚政治学、有特色的定性比较政治学）的研究评估的质疑。本文还介绍了2018年一项调查的发现，指出使用刊物排行对政治理论的研究有不利影响，对其它子领域则有积极影响。本文结尾提出了两个建议。

KEYWORDS:

• Political science
• political studies
• research evaluation
• journal rankings
• political theory
• Australia

Acknowledgments

I acknowledge the invaluable research assistance of Dr Gillian Stacey (research librarian at La Trobe University), who prepared the citational metrics to support parts of this analysis and who also compiled a bibliography on relevant research evaluation literature. I thank the many political theorists in Australia who provided responses to a long survey on this topic. I express my gratitude to four colleagues in political theory, including Professor John Dryzek and Dr Nicholas Barry, and also to one colleague in international relations, all of whom kindly provided feedback on earlier drafts. I have also benefited from discussions with a number of other colleagues, in political theory, international relations, and research management positions. Finally, I sincerely thank the Editors for AJPS and three anonymous reviewers for providing extensive feedback, across two rounds of review. Although the article has benefited enormously from this input, I am solely responsible for the content that remains.

Disclosure statement

No potential conflict of interest was reported by the author.

Notes on contributor

Miriam Bankovsky is a senior lecturer in Politics and the director of the Politics, Philosophy and Economics degree at La Trobe University in Melbourne. She has published Perfecting Justice in Rawls, Habermas and Honneth: A Deconstructive Perspective (Hbk Continuum 2012, Pbk Bloomsbury 2013), and two co-edited books on recognition theory and contemporary French philosophy (Manchester UP 2012, CNRS Editions Alpha 2012). In 2017, she received the Australasian Association of Philosophy's ‘Annette Baier Prize’ for her work on ‘Excusing Economic Envy’. Having completed an ARC DECRA fellowship in late 2018, she is now preparing a manuscript provisionally entitled The Family, Ethics and Mainstream Economics: An Unorthodox History. She has published work across philosophy, politics and economics, in journals that include Philosophy & Social Criticism, Journal of Applied Philosophy, History of Political Economy, and Cambridge Journal of Economics.

Notes

1. Although the Australian PSA is usually abbreviated to APSA, it is here referred to as AusPSA to distinguish it from its American equivalent, which also goes by APSA.

2. As submissions to the Working Party closed in June 2016, this analysis relies on 2015 citational metrics. Thomson Reuters normally provides a list for the previous year in the month of June. Scopus updates their lists bi-annually in April and September, again for the previous year, suggesting that if metrics were used, they would have been for 2015. SCImago relies on Scopus, and is likely to have been updated at the same time.

3. There are other gender-related issues with citation metrics. Evidence suggests some men are reluctant to cite women (Jump 2015). In the US, where larger proportions of women pursue qualitative methodologies (Vromen 2018, 240), citation-based measures may also disadvantage women relative to men. Sawer and Curtin (2016) have linked this to the increased disadvantage of women in US politics departments compared to Australian politics departments (which overwhelmingly prefer qualitative methodologies).

4. Garand (2005) uses correlation and factor analyses to argue that, in spite of sub-disciplinary and methodological differences in journal rankings, there remains a strong coherence in how political scientists from different subfields and methodological groups rank the full range of politics journals. But even if true at a disciplinary level, the use of disciplinary rankings in evaluative processes must still be viewed as problematic if it entails real disadvantage for certain sub-disciplines.

5. The 2011 ‘Teaching, Research and International Policy’ survey found 214 scholars in international relations in Australian politics departments. Using less inclusive criteria, Weller and Cowan (2012) counted 167 international relations/international political economy scholars. With numbers of scholars in Australian political science departments estimated at 400 in 2011 (Weller and Cowan 2012), these IR/IPE numbers represent between 42 per cent and 54 per cent of political studies hires, a proportion that will have increased since the count.

6. Bennett, Barth, and Rutherford (2003) suggest that what highly-cited US journals publish is not actually representative of the diversity of political studies in the US either. This article does not need to position itself on this interesting issue to make its own point.

7. The total within each ratio is greater than 100 per cent due to some articles using more than one method.

Additional information

Funding

Although I completed this article while employed as an Australian Research Council DECRA fellow (DE130100325) at La Trobe University, my research assistance costs were covered by a prior ECR Excellence in Research grant, awarded by La Trobe University.