Measuring research impact – does it matter?

This issue of the Australian Journal of Forensic Sciences (AJFS) marks the completion of volume 48 and our first year with six issues. As Editor I am pleased to relate that we have had no problem filling the six issues. Balancing author expectations for quicker publication against the reality of time taken to complete the review process remains a significant challenge but, overall, author experience feedback is very positive. I am also pleased to record that the impact factor for our journal increased to 0.833. This still leaves the AJFS in the lower third of forensic journals but we are moving in a positive direction. Impact factor is an important consideration for potential authors, especially from academic institutions; however, recent changes announced by the Australian Research Council (ARC) have, to some extent, changed the future weighting attached to journal impact factor by introducing a requirement for an ‘impact statement’ in ARC grant applications. The ARC defines research impact as ‘the demonstrable contribution that research makes to the economy, society, culture, national security, public policy or services, health, the environment, or quality of life, beyond contributions to academia’ (my emphasis). Researcher applicants will now be required to choose examples that demonstrate both the expected outcomes and benefits over time using a ‘research impact pathway’¹.

Australia is not alone in adopting such an approach, with the 2014 UK Research Assessment Framework (REF) also introducing an assessment of ‘research impact’ in addition to the quality of ‘research outputs’ and ‘research environment’. Impact was defined as ‘any effect on, change or benefit to the economy, society, public policy or services, health, the environment or quality of life, beyond academia’². In the first application of this new approach in the UK, 44% of impacts were judged outstanding and 40% were judged very considerable2. Of course, the changes to the UK system have not been without some criticism and controversy. It has been suggested that the definition of impact as ‘impact outside of academia’ interferes with academic freedom. Perhaps more persuasive are concerns about the challenges to measure impact in a fair and impartial way. Indeed, it has even been suggested, by no less than the Times Higher Education, that some universities appear to be ‘gaming’ the REF system including ‘REF poaching’ where universities recruit senior research staff with established research records immediately before the REF process³.

How easy is it to measure the impact of university research on society? According to Tim Cahill and Mark Bazzacco in the Conversation, not easy at all⁴. They specifically comment on the limitations of using case studies (the approach followed in the UK’s REF process) pointing out four main areas of concern as follows:

•	Timing – research often has impacts long after the research is completed.
•	Attribution – innovation is a team sport involving multiple projects, actors and inputs, often in isolation from each other.
•	Appropriability – it can be difficult to know who benefits from research and account for the diverse impacts that can arise.
•	Inequality – it is difficult to compare across different types of impacts. For example, how do you compare the development of Gardasil with research on the effects of climate change in the Murray-Darling Basin?

Cahill and Bazzacco also point out that whilst ‘a narrow focus on academic publishing has helped create a world-leading university sector’, measuring research by publications alone is not going to deliver a ‘system that can apply this research to improve the lives of the public’. They conclude that what is needed is a multi-dimensional set of measures that, importantly, do not ignore the need to do basic research.

A relatively recent issue of the journal Research Evaluation was wholly devoted to the issue of ‘impact’⁵. One contributor posed the question about the UK’s REF and the ‘impact agenda’, ‘are we creating a Frankenstein monster?’⁶. The author, Ben Martin⁶, drew attention to the complexity of assessing impact, including the cost involved in developing the case studies required by the REF process, arguing that, at some point, the graph of increasing compliance costs must inevitably cross the graph of diminishing benefits.

Whatever the challenges posed by measuring research impact it seems clear that it is here to stay. This does not mean that the importance of contributions to academic journals has suddenly disappeared. Research ‘quality’ will continue to use journal impact metrics but, perhaps, journal relevance in reaching the intended audience may assume a greater significance rather than blind adherence to journal impact factors. Notwithstanding, as the Editor of the AJFS, my future goals include continuing to seek ways to increase the impact factor for the AJFS to ensure our published authors continue to support our evolving journal.

James Robertson
Editor
[email protected]