Is external research assessment associated with convergence or divergence of research quality across universities and disciplines? Evidence from the PBRF process in New Zealand

ABSTRACT

Performance-based research evaluations have been adopted in several countries both to measure research quality in higher education institutions and as a basis for the allocation of funding across institutions. Much attention has been given to evaluating whether such schemes have increased the quality and quantity of research. This paper examines whether the introduction of the New Zealand Performance-Based Research Fund process produced convergence or divergence in measured research quality across universities and disciplines between the 2003 and 2012 assessments. Two convergence measures are obtained. One, referred to as β-convergence, relates to the relationship between changes in average quality and the initial quality level. The second concept, referred to as σ-convergence, relates to the changes in the dispersion in average research quality over time. Average quality scores by discipline and university were obtained from individual researcher data, revealing substantial β- and σ-convergence in research quality over the period. The hypothesis of uniform rates of convergence across almost all universities and disciplines is supported. The results provide insights into the incentives created by performance-based funding schemes.

KEYWORDS:

• Education policy
• performance-based research
• funding systems
• research quality
• convergence
• universities

JEL CLASSIFICATION:

• I2
• I28

Acknowledgments

We are grateful to the New Zealand Tertiary Education Commission for providing the data used here. An earlier version of this paper was presented at the New Zealand Association of Economists Conference in July 2019. We have benefited from the suggestions of an anonymous referee.

Disclosure statement

No potential conflict of interest was reported by the authors.

Supplementary material

Supplemental data for this article can be accessed here.

Notes

¹ Hicks (2012) defines the main characteristics of these systems as: (i) research is assessed, (ii) research evaluation is ex post, (iii) research output and/or impact is evaluated, (iv) part of government allocation of university research funding depends on the outcome of the evaluation, and (v) the scheme is a national or regional system. They vary by coverage and assessment methods, which may be based on bibliometric data or peer review.

² On the aims of the NZ PBRF see NZ Tertiary Education Commission (2002), Ministry of Education (2012), and Smart and Engler (2013).

³ The dollar amounts received were, however, weighted according to whether an academic discipline fell within one of three categories, as discussed in Section II.

⁴ As in other country studies, caution must be exercised in attributing causal impacts of such research assessment schemes. In the NZ case, there are no comparable data on pre-PBRF research quality. Further, since the scheme was introduced across all universities simultaneously, it is not possible to pursue a ‘treatment versus control’ approach.

⁵ The assessment and scoring method used in the New Zealand PBRF system from 2003 to 2012 are described in more detail and critically evaluated in Buckle and Creedy (2019b).

⁶ The recognition that new researchers may take time to establish their research, publications and academic reputations led to the introduction in 2006 of the new categories, C(NE) and R(NE), which applied to new and emerging researchers who did not have a full six-year portfolio. The following analysis does not distinguish the NE categories, since neither numerical scores nor funding was affected.

⁷ Since the grade for R-rated staff is zero, their number affects only the denominator in (2).:

⁸ This dataset is not publicly available. The TEC publishes summary results of each assessment round; see, for example, New Zealand Tertiary Education Commission (2013).

⁹ The number of staff who did not submit a portfolio (NP) do not enter into the calculation of AQSs derived in this paper because NPs cannot be identified by discipline. Including NP-staff in the Buckle and Creedy (2018, 2019a) confirm conclusions regarding the substantial increase in AQSs across all universities.

¹⁰ Only QCs, not component scores, are available. Also, the dataset does not include information on part-time status; hence $e_h$ = 1 was adopted in calculating AQSs. Values do not differ substantially from those reported by TEC.

¹¹ The amounts received also depended on the discipline to which a researcher belongs, with three separate categories given financial weightings of 1, 2 and 2.5; see Appendix A. Roa et al. (2009) estimate that, following the 2006 PBRF, universities received $34,166 per year per A-researcher (discipline weighting = 1) and $6,832 per year for each C-researcher in the same discipline. These weightings remained the same throughout the period.

¹² Indirect financial effects on universities could result from research assessment schemes such as the PBRF. For example, Biancardi and Bratti (2019) find that universities that performed better in the Italian Research Evaluation Exercise benefited from more students with higher entry qualifications, with effects stronger for top performing universities.

¹³ Writing $z=q/\mu$, if only random proportional changes in the ratio, z, occur, then $(dz/z)dt=u$ which in discrete time converts to $log{z}_t-log{z}_{t-1}=u_t$. To allow for relative size to have a systematic contribution, add a term, $\beta log{z}_{t-1}$ to the right-hand side. Hence, when $q_{t-1}/{\mu }_{t-1}<1$, so that q is initially below the geometric mean, the logarithm of the relative value, z, is negative. Hence if $\beta <1$, the proportional change is positive and larger than when q is initially above the geometric mean.

¹⁴ For example, if β = – 0.7 the stable variance, ${\sigma }_q^2$, is about 10% higher than ${\sigma }_u^2$, but if β = – 0.5 the difference rises to 33%, and when β = – 0.2 the difference becomes 178% higher (and 426% higher when β = – 0.1).

¹⁵ As shown in Table 1, Lincoln University has no observations for Education and Law, so that the total number of universities and disciplines shown in Table 2 is 70 rather than 72.

¹⁶ Buckle and Creedy (2019a) show that AQS improvements involved all universities reducing the proportion of the lowest QC researchers (R-rated) and increasing A- and B-rated staff. Only VUW reduced, rather than increased, the share of C-rated researchers.

¹⁷ Random shocks in this case might include, for example, the 2010–11 Canterbury earthquakes which may have affected the ability of Canterbury and Lincoln to recruit staff after 2011. In NZ’s small university system this may have indirectly affected recruitment practices, including other universities recruiting higher-quality staff from CU and LU.

¹⁸ Regressions in Table 3 use 87 observations: 70 for the 8 universities and 9 disciplines within universities, less two missing observations for Law and Education at Lincoln, plus 17 AQS_ij values, averaged across all universities and all disciplines. This enables parameters for each university’s growth and convergence to be compared directly with the average across all universities or disciplines rather than adopting one university and discipline as the omitted variable. However, adjusted-R²s must be interpreted cautiously since they are somewhat inflated by the inclusion of individual observations and their cross-university or cross-discipline averages. For example, running the regression specification in Table 3 on the 70 AQS_ij observations yields a slightly lower adjusted-R² = 0.940.

¹⁹ As mentioned earlier, both CU and LU experienced severe infrastructure damage from the 2010–11 Canterbury earthquakes. However, there are no significant shift dummy effects identified for CU, suggesting, it is hard to argue that an earthquake effect fully explains the LU shift.

²⁰ These data are reported in Appendix A, Tables A1 and A2.

²¹ Results obtained for non-log versions of these included variables, and when mean, rather than median, age was used were uniformly inferior to those reported in Table 4, with no variables even close to statistical significance at the 10 per cent level.

²² Testing whether the shift and slope dummy variables included in Table 3 become redundant when new variables are added to regressions confirms that this is not the case. A nested model always supports the inclusion of all dummy variables over the age or scale variables.

²³ From Table 4, the total convergence effect is given by: d(AQS growth)/d(logAQS₂₀₀₃) = –0.796 + 0.016*logN₂₀₀₃ = –0.796 + 0.074 = –0.722, using mean logN₂₀₀₃ = 4.638.