The evolution of research quality in New Zealand universities as measured by the performance-based research fund process

ABSTRACT

This paper examines how research quality of New Zealand university staff has evolved since introduction in 2003 of the Performance-based Research Fund (PBRF). The analysis uses a database consisting of an anonymous ‘quality category’ (QC) for each individual assessed in each of the three PBRF rounds. Emphasis is on evaluation of organisational changes. The paper examines the extent to which each university's average quality score (AQS) changed as a result of changes in QCs of existing staff over time and from staff exit and entry. The data also include information about the age of staff evaluated in PBRF. This is used to assess changes in the age distribution of researchers across universities, and ages of those making transitions within universities and between grades. A number of hypotheses regarding organisation change in response to the introduction of PBRF are discussed and tested by comparing universities with different patterns of change.

KEYWORDS:

• Education
• New Zealand universities
• performance-based research fund
• research

View correction statement:

Correction

Acknowledgments

We are grateful to Amber Flynn and Leon Bakker for assisting engagement with New Zealand Tertiary Education Commission (TEC), and TEC for providing the anonymous raw PBRF data requested by the authors. We have benefited from discussions with Sharon Beattie, Shelly Biswell, Jonathan Boston, Norman Gemmell, Gary Hawke, Morgan Healey, Adam Jaffe, Julie Keenan, John MacCormick, Katy Miller and Grant Scobie. We should also like to thank Matthew Ryan and two referees for their constructive suggestions. An earlier version of this paper was presented at the New Zealand Association of Economists Conference, Wellington, July 2017.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. See http://www.tec.govt.nz/funding/funding-and-performance/funding/fund-finder/performance-based-research-fund/. See also Mahoney (2004) and Smart and Engler (2013).

2. Since 2014, the UK RAE scheme has been known as the Research Excellence Framework (REF). On other schemes, see, for example, Bakker, Boston, Campbell, and Smyth (2006), and Ministry of Education (2013a). On the Australian exercise, see Roberts (2005). For a summary of international approaches, see Coryn (2007), Jones and Cleere (2014) and de Boer et al. (2015).

3. There are variations in the extent to which universities depend on public funds; this may affect the strength of incentives created by PBRF.

4. Wider aspects of the PBRF process have been raised by other authors, but cannot be examined here. These include, for example, the effect on university priorities by Walsh (2004); concerns about academic freedom and unintended distortions, by Curtis (2004), Boston, Mischewski and Smyth (2005), Curtis and Matthewman (2005), Duncan (2008), Ministry of Education (2008), Whitley (2008), OECD (2010), Clear and Clear (2012), Butler and Mulgan (2013), Chavarro, Tang, and Ràfols (2016); and the need to consider costs by Hazledine and Kurniawan (2005).

5. Since 2016 the percentage allocations have changed to 55:25:20 respectively.

6. These categories are described further in Ministry of Education (2012, p. 21). See also New Zealand Tertiary Education Commission (2016).

7. TEC have also produced a range of measures of AQS which vary according to the choice of denominator, including the sum of effective full-time students and the sum of postgraduate students. These are affected by the discipline mixture and in particular by student/staff ratios, and hence do not necessarily reflect average research quality of staff.

8. However, where individuals moved within a year of the census date, the funding was divided between the two relevant institutions.

9. The staff numbers for the Universities of Auckland, Canterbury, Otago and Victoria University of Wellington used in this paper include the staff from these respective Colleges of Education.

10. The role of reputation effects is stressed in OECD (2010).

11. They range from 0.01 (or 0.3% of the TEC AQS for Massey in 2006) to 0.46 (or 12% of the TEC AQS for VUW in 2006).

12. This result complements the finding of Gemmell, Nolan, and Scobie (2017) that research productivity in NZ universities has increased markedly since the early 2000s.

13. Similarly, $AQ{S}_{N_T}=(AQ{S}_{(-R)}\left(N_P-N_{(R)}\right)/N_T$.

14. As mentioned in the note to Table 1, the number of total non-administration staff for VUW in 2006 included part-time staff and tutors in the academic staffing category who did not meet the TEC criteria for inclusion, and hence the AQS based on all staff is lower than it should have been.

15. The 2012 round took place after the 2011 severe Canterbury earthquakes. TEC agreed on a number of actions to mitigate the effect on academic staff at Canterbury and Lincoln universities: see New Zealand Tertiary Education Commission (2013b).

16. Several authors have endeavoured to assess the impact of PBRF on academic research by comparing publication measures before and after the introduction of PBRF: examples of this type of work as applied to the economics discipline include Gibson, Anderson, and Tressler (2008), Anderson, Smart, and Tressler (2013), Anderson and Tressler (2014). For examinations of a wider range of disciplines, see Hodder and Hodder (2010) for New Zealand, and Butler (2003) for Australia, who concluded that research quantity increased but quality declined. A critique of Butler is provided by van den Besselaar, Heyman, and Sandström (2017), who find that both quantity and quality of research has increased, and refer to studies showing a positive relationship between research quantity and impact.

17. It is not possible to know the age at which transitions are made, as only the age at the time of a portfolio submission is known.

18. In comparing performance relating to the exits of R-researchers, it has to be kept in mind that this figure is distorted for reasons discussed in the introduction and in Section 3.

19. See https://www.studyinaustralia.gov.au/global/australian-education/universities-and-higher-education/list-of-australian-universities, https://en.wikipedia.org/wiki/List_of_universities_in_the_United_Kingdom, http://learningenglish.voanews.com/a/a-23-2005-05-11-voa1-83125492/124600.html and https://en.wikipedia.org/wiki/List_of_universities_in_Canada.

20. This point was also made by the Tertiary Education Commission (2013a, p. 66).