The ‘disciplinary effect’ of the performance-based research fund process in New Zealand

Abstract

This paper examines how the research quality of academic disciplines within New Zealand universities has evolved since the Performance-based Research Fund (PBRF) began in 2003. It uses a database consisting of an anonymous ‘quality category’ (QC) for each person in the 2003 and 2012 assessment rounds. Individuals are assigned to nine discipline groups and the paper measures the distribution of researchers across disciplines within universities. There has been little change in the distribution and their concentration within and across universities. Exceptions are increases in the shares of medicine and agriculture, and a reduction in the share of education. Average Quality Scores are derived for each discipline. All groups substantially increased their scores. Transition matrices show that there are significant differences in the dynamics of disciplines during the PBRF process. Changes in the discipline composition of universities explains little of the proportional improvement of research quality among New Zealand universities.

KEYWORDS:

• Academic disciplines
• education policy
• New Zealand universities
• Performance-Based Research Fund
• productivity
• research
• transitions

Acknowledgements

We are grateful to Amber Flynn for assisting with engagement with New Zealand Tertiary Education Commission (TEC), and TEC for providing the data. We have benefited from discussions with Sharon Beattie, Morgan Healey and Shelly Biswell, and comments by Norman Gemmell and two referees on an earlier draft of this paper.

Disclosure statement

No potential conflict of interest was reported by the authors.

ORCID

Robert A. Buckle http://orcid.org/0000-0002-8708-0669

John Creedy http://orcid.org/0000-0001-5061-0075

Notes

1 See https://www.tec.govt.nz/funding/funding-and-performance/funding/fund-finder/performance-based-research-fund/. See also New Zealand Tertiary Education Commission (2002), Mahoney (2004), Ministry of Education (2012), and Smart and Engler (2013).

2 The dataset is not publicly available. A fourth round was completed in 2018 but the results for that round were not available at the time the research for this paper was undertaken. The present analysis extends the earlier work of Buckle and Creedy (2017, 2018a, 2018b, 2019) which evaluated the evolution of research quality in each New Zealand university between the 2003 and 2012 PBRF rounds, and assessed the metrics used in those processes. However, overall university changes were considered in those papers without any disaggregation by discipline group.

3 Buckle and Creedy (2018b) provide a detailed explanation and critique of the method used to assess the research quality of individual researcher’s portfolios in the New Zealand PBRF system and the metrics used to rank and compare universities in the 2003, 2006 and 2012 rounds.

4 This result is consistent with the finding of Gemmell, Nolan, and Scobie (2017) that research productivity in NZ universities has increased markedly since the early 2000s.

5 This may be a consequence of the self-reinforcing nature of the funding formula used by TEC which reduces the effective price of A-researchers for those universities which are already better placed to attract top quality researchers, and which increases the relative marginal quality improvement of recruiting lower-quality faculty by lower-ranked universities; see Buckle and Creedy (2018a, pp. 13–15 and Figure 3).

6 Information about The Changing Academic Profession project is available from: https://melbourne-cshe.unimelb.edu.au/lh-martin-institute/research/projects/the-changing-academic-profession

7 The period since 2003 coincides with changes in Government tertiary education policy with respect to former Colleges of Education, and there were significant changes in Education staffing levels.

8 The assessment and scoring method used in the New Zealand PBRF system for the 2003, 2006 and 2012 rounds are described in more detail and critically evaluated in Buckle and Creedy (2018a, 2018b, 2019).

9 The recognition that new researchers may take time to establish their research, publications, and academic reputations led to the introduction in 2006 of the new categories, C(NE) and R(NE). These categories applied to new and emerging researchers who did not have the benefit of a full six-year period. The following analysis does not distinguish the NE categories, since numerical scores are not affected.

10 The employment weight per researcher, $e_i$ is not available from the data set used in this study and hence in deriving each discipline AQS, $e_i=1$ for each portfolio (that is, for each i). In the calculation of university AQSs, Buckle and Creedy (2019, section 4.1, Table 3) show that employment weighting makes only a small difference to the calculation of the university-level and New Zealand total AQSs and the ranking of university AQSs.

11 See Buckle and Creedy (2018a, pp. 29–30).

12 The changes are: Law (41 per cent), Core Science (40 per cent), Humanities (32 per cent), Education (19 per cent), Management (16 per cent), Engineering (15 per cent), AFE (10 per cent).

13 This table is taken from Buckle and Creedy (2019, Table 8).

14 Details about movements from one NZ university to another NZ university were not available. Hence, they appear in the table simply as being among the exits and entrants.

15 However, as explained in Buckle and Creedy (2019), the proportions for exits are somewhat misleading because some of the individuals could simply have been given new contracts which meant that they avoided the need to submit a portfolio.

16 Details about movements from one NZ university to another NZ university were not available. Hence, the exits and entrants for discipline groups are likely to be overstated somewhat.

17 Summary measures of turnover are presented in Appendix 2.

18 In comparing performance relating to the exits of R-researchers, it must be kept in mind that this figure is distorted for reasons discussed in Section 4.2 above.

19 Further applications of the method can be found in, for example, Bargain (2012), Creedy and Hérault (2015), Ball and Creedy (2016) and Creedy (2017).

20 Buckle and Creedy (2019, p. 4, Table 1) report information about the number of full-time equivalent portfolios, NP, and non-administrative staff, NT. In the calculations here, the number of staff for whom a portfolio was not submitted in 2012 is approximated by NT-NP.