![Sample our Area Studies journals, sign in here to start your access, latest two full volumes FREE to you for 14 days]({"type":"image" , "src" : "/sda/5922/TOC-Subject-Sample-2021-Area-Studies.jpg"})
Abstract
National research evaluation systems that use metrics for the assessment of academic institutions are usually regarded as exemplifying the same neoliberal model of governance that, with minor variations, is implemented worldwide. This essay argues, however, that despite apparent similarities, metrics are used for different aims in different national cases. It compares the use of figures in the UK RAE/REF, a prototypical neoliberal framework, with various schemas of assessment that have been used to evaluate Russian universities in recent decades. It argues that in the RAE/REF, the principal role of statistics is to solve ‘the lazy agent’ problem by creating a prisoner's dilemma for academic institutions, while in the Russian case, statistics serve to solve ‘the corrupt knower’ problem, preventing collusion between the assessor and the assessed. The essay concludes by putting forward some hypotheses on the origins of different approaches to quantification.
Notes
1 See reviews in Espeland and Stevens (Citation2008), Lamont (Citation2012), Berman and Hirschman (Citation2018).
2 In presenting his phenomenological sociology, Schutz famously drew a distinction between first-order constructs, which are used by agents themselves, and second-order constructs developed by an analyst (Schutz Citation1962). To have explanatory power, second-order constructs must take first-order constructs into account. A few authors, from Keynes to Robert Nisbett noted, however, that in reality the distinction is blurred; on the one hand, theorists rely on their pre-theoretical intuitions and, on the other, because second-order theories may eventually achieve first-order status. In a later section, I argue that ‘bad governance’ theories have largely functioned as first-order constructs throughout Russia's academic history and are responsible for many of its distinctive traits (Gel’man Citation2017).
3 A comprehensive overview of research policies of the Russian state during the ‘research turn’ is still lacking. The beginning of this ‘research turn’ is documented by Graham and Dezhina (Citation2008). Some of the later developments are discussed in Dezhina (Citation2020). There is also a considerable literature dealing with the effects of the ‘5–100’ excellence programme (see below)—for example, Poldin et al. (Citation2017) and Matveeva et al. (Citation2021). Possibly the best source on it is a recent report by the Audit Chamber of Russian Federation, Byulleten' Schetnoi palaty RF. Universitety, 2021, available at: https://ach.gov.ru/upload/iblock/845/845aaecb7eee3453e759d3c52a761bda.pdf#page=3, accessed 28 February 2021.
4 ‘Vuzy prosyatsya na zarabotki. Dmitrii Medvedev provel zasedanie rektorov vuzov’, Kommersant’’, 21 March 2007, available at: https://www.kommersant.ru/doc/751671, accessed 28 February 2021.
5 Ukaz Prezidenta Rossiiskoi Federatsii 'O realizatsii pilotnogo proekta po sozdaniyu natsional'nykh issledovatel'skih universitetov', 28 October 2008, available at: http://web.archive.org/web/20100429031342/http://mon.gov.ru/dok/ukaz/obr/4949/, accessed 28 February 2021.
6 The ‘centres of excellence’ approach to funding of research at universities consists in allocating block grants to HEIs (or their divisions) submitting development proposals and winning in a competition. The major alternative to it is ‘formula funding’, in which available money is distributed among all eligible institutions proportionally to their performance in the previous period. RAE/REF is the best-known example of the later approach.
7 The purchasing power of this assignment decreased due to the devaluation of the Russian currency in 2014–2015 from approximately $30 million to $16 million.
8 As an example, see the ‘roadmap’ of St Petersburg Electrotechnical University, available at: https://etu.ru/assets/files/university/roadmap.pdf, accessed 28 September 2019.
9 Byulleten' Schetnoi palaty RF. Universitety, 2021, available at: https://ach.gov.ru/upload/iblock/845/845aaecb7eee3453e759d3c52a761bda.pdf#page=3, accessed 28 February 2021.
10 During the first round of the ‘5–100’ programme, the Ministry of Education and Science announced that it planned to select 15 winners (‘15 rossiiskikh vuzov poluchat gospodderzhku dlya vkhozhdeniya v mirovye reitingi’, Kommersant’’, 8 July 2013, available at: https://www.kommersant.ru/doc/2229575, accessed 21 February 2021). However, during the second round (2015) six more were added to this list.
11 Indirectly, of course, universities were competing with each other, as the amount of money available was probably fixed, and each university scoring above the threshold thus decreased the sum other winners received. The mechanics of this competition were never publicly discussed, however.
12 The state funding of Russian public universities is comprised of two unequal parts. The first and greater consists of state subsidies covering the costs of teaching certain numbers of students (so-called byudzhetnye mesta, state-funded places). Until the early 2000s, a university received funding for teaching a certain number of students and could decide for itself how to allocate these subsidies between different departments and specialties. Since the start of the ‘research turn’, state-funded places have been tied to a given subject. The second part of public funding arises from various target programmes such as National Research Universities or ‘5–100’. Even the most lavishly funded of such programmes, ‘5–100’, gives its participants around 10% of their total income and except for a handful of elite winners, all schools are still predominantly funded through tuition and subsidies tied to budzhetnye mesta as the only significant source of revenue (Sokolov Citation2017).
13 In each particular case, the government's actions may be regarded as opportunistic reactions. Thus, shutting down private universities after 2014 was possibly a way of channelling tuition-paying students to the public universities and thus supporting their faculties when the real income of public sector workers fell. The cancellation of the student voucher experiment was most likely an attempt to avoid the mass migration of student youth from the regions to the capital cities. Nevertheless, overall, we find a situation that is the opposite of prototypical New Public Management reforms.
14 For example, those applying for RSF grants were required to have at least five articles in journals indexed by Scopus. Applicants for major grants or those considered for inclusion in expert bodies are also to provide, in addition to simple publication counts, numbers of citations, Hirsh indexes and even more elaborate scientometric measures (such as ‘cumulative impact factor’). The use of such complex indicators negates an alternative explanation for the bureaucratic obsession with scientometrics: that this is simply a means of boosting a federal university's standing in the world university rankings. If so, there would be no need to invent new metrics that are not implemented in the existing rankings.
15 Indeed, some of the measures used by UK bureaucrats in assessing a university's performance showed a naïve faith in the trustworthiness of scholars as evaluators. Thus, for a period of time, UK universities could improve their standing in the government's eyes and win additional funding proportional to the number of highest grades awarded to their graduates. Their Russian counterparts would find it hilarious that nobody foresaw the rapid grade inflation.
16 Needless to say, this schema is prone to many kinds of mistakes. Most immediately, it depends on A's understanding of the meaning of citing for numerous Cs at the first stage, at which they choose to cite or not to cite. For science, see the numerous criticisms of citation measures pointing to the heterogeneity of citing practices across specialties (for example, in sociology and microbiology, citation means different things; Hargens Citation2000). The further problems it gives rise to are discussed in a later section.
17 The sociological case was particularly notorious and, possibly, heavily influenced the thinking of the decision-makers. Around 2006–2012, six nation-wide sociological associations were active in Russia, largely divided across political lines. Their feuds resulted, among other things, in two rival national sociological congresses being held at the same time in October 2012, one in Moscow, the other in Ufa. Two of the more conservative associations were under the control of self-proclaimed Orthodox sociologist Vladimir Dobren’kov, the Dean of the Moscow State University sociological faculty. Dobren’kov also controlled the sociological Curriculum Committee (UMO), using its powers to accredit sociological study programmes to control his supporters. While ministerial bureaucrats might tolerate or even sympathise with Dobren’kov's political anti-Westernism, they probably understood that his and similar figures’ control over Russian social sciences could hardly lead to an increase in their international visibility. Dobren’kov was also accused of multiple episodes of petty corruption, and extensive plagiarism was discovered in textbooks authored by him (Titarenko & Zdravomyslova Citation2017, p. 108).
18 Excerpts from a transcript of the meeting of Andrei Fursenko with faculty of the European University at St Petersburg, 24 February 2014.
19 In principal–agent literature, these longer chains are usually believed to be reducible to elementary dyadic ones. A criticism of this view arising from the study of scientific institutions can be found in Braun (Citation1993).
20 From Viktor Glukhov's (RISC) response to a question by the author at the roundtable ‘Rossiskie i zarubezhnye indeksy nauchnogo tsitirovaniya’, European University at St Petersburg, 3 April 2018.
21 For a sample of such arguments, one can read a collection of letters from academic councils of various social-scientific institutes of the Russian Academy of Sciences which protested against the introduction of a method of calculating organisational performance indicators using international citation indexes in 2020 (‘Obsuzhdenie metodiki rascheta kompleksnogo balla publikatsionnoi rezul'tativnosti’, 11 March 2020, available at: www.sib-science.info/ru/ras/predlozhennaya-09022020, accessed 28 February 2021).
22 The Russian government also considered employing international reviewers in the assessment of applications for major grants. Grant proposals for the RSF and megagranty must be submitted in both Russian and English. As can be deduced from the language of reviews made available to applicants, so far, the great majority of reviewers are Russian. It is not clear if the reliance on Russian reviewers is mostly for organisational and financial reasons or because foreigners are suspected of readiness to steal the nation's scientific secrets.
23 Dissernet also discovered that all sorts of academic dishonesty was particularly likely to be found in dissertations by politicians and bureaucrats themselves. It seems that university rectors and other members of Russian academic establishments often organised the defences of fraudulent dissertations by high-profile political figures, partly for the sake of personal enrichment, partly for seemingly more benign reasons such as securing political patronage for their institutions. Whatever the motives, the personal experience bureaucrats had in dealings with the academic establishment could hardly make them trust the latter's integrity.
24 Here, recent studies comparing amounts of doctored data in scientific papers from different countries (Fanelli et al. Citation2017) show great promise. Most likely it will soon be possible to draw comparisons regarding the levels of academic dishonesty on a much firmer basis.
25 I am grateful to the anonymous reviewer who suggested the example of Dissernet. The same reviewer also pointed that, while cases of politicians being caught plagiarising are in no way limited to Russia, the public reactions to them are different. Zu Guttenberg lost not only his degree, but also his post as the Minister of Defence in Merkel's government as a result of unauthorised borrowings being discovered in his PhD. The discovery of borrowed text in Vladimir Putin's dissertation had no repercussions for him (‘Researchers Peg Putin as a Plagiarist over Thesis’, The Washington Post, 24 March 2006, available at: https://www.washingtontimes.com/news/2006/mar/24/20060324-104106-9971r/, accessed 28 February 2021). The point is, however, that copying several pages of a literary review, discovered in Putin's dissertation, in no way discredits him in the eyes of most Russian academics. While there are probably good reasons to think that a more stringent approach to textual authenticity would be beneficial for a scientific community, the fact is that Putin did not break the operating norms of the community that judged his work.
26 On self-enforcing institutions, see Greif and Laitin (Citation2004).
27 Evaluation of the effects of the RAE falls completely beyond the scope of this essay. Critics of the RAE/REF were quick to note that it had wide and not entirely beneficial influences (McNay Citation2003). The RAE was accused of leading to the reallocation of resources and disadvantaging multidisciplinary and professional fields, as well as large-scale research requiring extensive time periods and not leading to fast publication (Whitley et al. Citation2018); advantaging already privileged institutions and concentrating wealth in a handful of elite schools (Boliver Citation2015); and stimulating large-scale gaming, including, above all other things, the practice of hiring international stars for short-term positions. All of these effects are also arguably present in Russia. In addition to that, however, other effects are observable that seem even more extensive and, at times, damaging.
28 In fact, state agencies themselves participated in attempts to improve bibliometric profiles of Russian academia as a whole by means quite close to what can be described as gaming. At the time this essay was being written, the Russian Foundation for Fundamental Research announced a competition for the support of review articles to be published in Russian journals included in Scopus. Each of the papers is to include no fewer than 50 references. While it is not stated that most of these sources are to be Russian, that seems quite explicit. The aim of the competition is openly described as ‘increasing international visibility of Russian science’ (‘Konkurs na soiskanie finansovoi podderzhki dlya podgotovki i opublikovaniya nauchnykh obzornykh statei’, Russian Foundation for Fundamental Research, 28 April 2020, available at: https://www.rfbr.ru/rffi/ru/contest/n_812/o_2105921, accessed 28 February 2020).
29 Professor, plate-glass university, 30 October 2010.
Additional information
Notes on contributors
Mikhail Sokolov
Mikhail Sokolov, European University at St Petersburg, 6/1A Gagarinskaya Street, 191187 St Petersburg, Russian Federation. Email: [email protected]