Italy’s national research assessment: some unpleasant effects

ABSTRACT

Higher education institutions are in a period of profound transformation. In particular, many governments have implemented mechanisms that attempt to relate funding to performance. In this article, we reflect about the relevance of those changes from a regional point of view, by analysing the recent introduction of a performance-based research funding system in Italy. The findings lead to policy indications, questioning the effectiveness of selective funding of universities based on national research assessment exercises when there is heterogeneity across regions in universities scientific performance and, at the same time, there is a huge dispersion of research performance within the individual universities.

KEYWORDS:

• Performance-based research funding systems
• research evaluation
• higher education
• regional development
• Italy

Acknowledgments

The authors thank two anonymous referees for their valuable comments.

Data availability

Data are available from the Italian Ministry of Education, Universities and Research (MIUR) and from the National Agency for Evaluation of the University System and Research (ANVUR).

Disclosure statement

No potential conflict of interest was reported by the authors.

ORCID

Francesco Prota http://orcid.org/0000-0001-6251-6916

Notes

1 The ‘New Public Management’ is a general public policy agenda that tries to increase efficiency and value for money in the public domain (Pollitt 2003; Pollitt and Bouckaert 2011).

2 On the possible drawbacks in the adoption of competitive and evaluation-based policies see the interesting debate on the effects of the Australian performance-based research funding system in the special section of the Journal of Informetrics 11, number 3 (2017).

3 This assumption is highly questionable in terms of both efficiency and equity. As shown by Moed et al. (2011) there is no evidence that more concentration of research among a country’s universities or among an institution’s main fields is associated with better overall performance.

4 All acronyms used in this paper are based on the original (Italian) denomination, while the expanded forms are in English.

5 A first wave of reforms commenced in 1989. In 1994, the mechanism of public funding was radically reformed.

6 The next section will describe in detail the evaluation procedure.

7 Italy is characterized by a central government funded system that relies mainly on a single grant, the Ordinary Financing Fund, for teaching, research and other infrastructural needs. The Ordinary Financing Fund is in the form of a lump-sum budget, which is managed autonomously by each university. Other sources of funding are student fees and contractual funding from MIUR and from other organizations. The importance of other contractual funding varies across regions, with the share received by universities in northern Italy being almost twice that received by universities in the central or southern parts of the country.

8 Four are the main differences. Firstly, the VTR is a peer-review type exercise, while the VQR either adopts peer review or citation analysis or a combination of the two. Secondly, the two evaluation exercises differ in the ratio of products to be presented by each university: one every four researchers belonging to each disciplinary areas in the case of VTR and three for each university researcher in the case of VQR. Thirdly, the time span of the evaluation exercise increases from three to seven years. Fourthly, the scale for rating individual research products is different. A new assessment exercise, denominated VQR 2011-2014, has been recently implemented. Its architecture is rather similar to that of the previous VQR. For a comparison of the results of these two evaluations (the VQR 2004-2010 and the VQR 2011-2014), from a territorial point of view, see Prota and Grisorio (2017).

9 The 14 areas are: Mathematics and informatics (Area 1); Physics (Area 2); Chemistry (Area 3); Earth Sciences (Area 4); Biology (Area 5); Medicine (Area 6); Agricultural and Veterinary Sciences (Area 7); Civil Engineering and Architecture (Area 8); Industrial and Information Engineering (Area 9); Antiquities, Philology, Literary studies, Art History (Area 10); History, Philosophy, Pedagogy and Psychology (Area 11); Law (Area 12); Economics and Statistics (Area 13); Political and Social sciences (Area 14).

10 Excellent (score 1): the research work is placed in the top 20 percent of the ‘‘scale of values shared by the international community’’; Good (score 0.8): the research work is placed in the 60–80 percent interval of the scale; Acceptable (score 0.5): the research work is in the 50–60 percent interval of the scale; Limited (score 0): the research work is in the lowest 50 percent of the scale.

11 Area weights are determined on the basis of the share of research outcomes submitted in each area out of the total outcomes of the university.

12 The R² of the regressions for the share of excellent research outputs in total expected research outputs and the share of excellent and good research outputs in total expected research outputs shows that these models are able to explain about 60% of the variability of the data.

13 Abramo, D’Angelo, and Rosati (2016) measure the research performance at the individual and aggregate levels by means of a proxy of labour productivity, using field-normalize citations and the fractional contributions of scientists to outputs, over the period 2009–2013.

14 In this regard, a recent paper by Prota and Grisorio (2017) compares the results of the Italian research evaluation assessment for the period 2004-2010 and for the period 2011-2014 from a territorial perspective. Their results show that there is not a clear evidence of convergence among Italian macro-regions in terms of universities performance and that the variability of performance within each Italian university has increased in the VQR 2011-2014 compared to VQR 2004-2010.

Additional information

Funding

The authors received no specific funding for this work.