Assessing research impact potential: using the transdisciplinary Outcome Spaces Framework with New Zealand’s National Science Challenges

ABSTRACT

Calls for science to have impact as well as excellence have been loud and clear from research funders, policymakers and research institutions for some time. Transdisciplinary research (TDR) is expected to deliver impact by connecting scientists with stakeholders and end users to co-produce knowledge to respond to complex issues. While New Zealand’s science system is geared to deliver excellence, its capability to also deliver impact beyond academic institutions is less clear. This paper has two interconnected aims. Firstly, it presents findings from testing innovations to the TDR Outcome Spaces Framework (OSF+) with four National Science Challenges (NSCs). We conclude that OSF+ is a useful tool for planning for multiple outcomes and assessing the potential for impact. Secondly, it presents findings of how using OSF+ to assess research impact potential revealed a range of implicit theories of change (i.e. catalyst, deficit, engagement and collaboration) across the NSCs. The findings raise important questions about the prospects for New Zealand’s science system to deliver the envisaged and needed levels of research impact when current institutional settings, expectations, recognition systems, career paths and measures of success are not yet able to adequately accommodate TDR to deliver the research impact.

KEYWORDS:

• Outcome Spaces Framework
• National Science Challenges
• theories of change
• research impact
• transdisciplinarity

Introduction

Transdisciplinary research (TDR) and practice are characterised by the involvement of multiple disciplines of science alongside end-users and stakeholders in collaborative research to address real-world sustainability issues (Mitchell et al. 2015). It is these features that are recognised by research funders, policymakers and research institutions as both necessary for delivering research impact, and essential for addressing the complex issues societies now face (Gibbons 1994; Roux et al. 2010; Hansson and Polk 2018; Schneider et al. 2019). However, doing TDR takes considerable time and effort. There are few rewards for practitioners, and it can be as messy and complex as the issues it seeks to address. Furthermore, evaluating its progress or effectiveness can be challenging given that its experiences and benefits are often intangible or broadly distributed across multiple actors, sectors and scales (Wickson et al. 2006; Mobjörk 2010; Roux et al. 2010; Polk 2015; Di Iacovo et al. 2016; Holmes et al. 2018; see also MBIE 2017, 2019a).

Excellence and impact are the foundations of New Zealand’s science system (MBIE 2015, 2017). The New Zealand Government’s National Statement for Science Investment 2015–2025 (NSSI) defines excellence as ‘well-designed, well-performed, well-reported research, recognised as such, eg through peer review’ (MBIE 2015, p. 6). It defines impact as ‘the direct and indirect “influence” of research or its effect on an individual, a community, or society as a whole, including benefits to our economic, social, human and natural capital’ (MBIE 2015, p. 6). However, the societal impact that research is expected to deliver is difficult for research funders and institutions to recognise when institutional settings, reward and recognition systems, and measures of success have been established to advance science through disciplinary and specialised research (Thompson et al. 2017; West et al. 2019) and to deliver academic rather than societal impact (MBIE 2015).

Linking science excellence and impact: the challenges

It has been acknowledged within the NSSI that change is needed in the assessment of impact in order to go beyond academic impact and its measures of success (MBIE 2015). Work on this issue has recently culminated in MBIE’s (2019a, p. 1) The Impact of Research: Position Paper, which now defines research impact as a ‘change to the economy, society or environment, beyond contribution to knowledge and skills in research organisations’ (see also MBIE 2017). While this is important policy direction, how we go about doing research for impact remains unclear and the institutional structures that foster and assess impact, as an extension of the existing criteria for science excellence, still have the potential to significantly constrain TDR in practice (Turner et al. 2016; Hansson and Polk 2018).

One part of the problem in New Zealand, and elsewhere, is that science funding systems embody a knowledge deficit theory of change (Irwin and Wynne 1996; Jasanoff 2004; Turner et al. 2016; Oberlack et al. 2019; West et al. 2019), whereby science is assumed to be both the keyhole and the key to change. In other words, the persistence of social-ecological problems is assumed to originate from a lack of (usually biophysical) science, and filling science gaps or the provision of science to those assumed to be the problem is expected to, somehow, instigate action (Hackmann et al. 2014; Fernandez 2016; West et al. 2019). This theory of change enlists science communicators, boundary spanners and knowledge brokers, who are tasked with packaging and translating science to make it more usable or visually appealing (West et al. 2019). This ‘linking knowledge to action’ paradigm conceives of the ‘lack of uptake and impact’ as residing in how science is communicated and packaged rather than in the science system itself (see Turner et al. 2016; West et al. 2019). Given the challenges societies now face, this modus operandi is increasingly recognised as unable to deliver societal impact (Turner et al. 2016; Leith et al. 2017; West et al. 2019; Wyborn et al. 2019). Hence the calls for TDR.

The TDR theory of change involves co-design, co-production and research collaboration, which require unique skills (Botha et al. 2017; Fam et al. 2017; Djenontin and Meadow 2018; West et al. 2019; Wyborn et al. 2019) that disciplinary science structures and settings often fail to recognise and value (Turner et al. 2016). The NSSI envisages excellence and impact as deriving from ‘the best people’, ‘a rigorous approach’ and ‘optimum results’ (MBIE 2015, p. 13). While these might be useful principles for assessing science excellence, their relevance for assessing impact is limited. For example, TDR practitioners in New Zealand have found they must do far more than communicate or package science to make it relevant for societal action. When stakeholders and Māori partners are involved, researchers face legitimacy challenges: what is useful and credible for one audience is often not the case for others. Knowledge brokers need to walk multiple paths with multiple actors to co-produce knowledge that has utility and legitimacy with multiple audiences. This takes time, empathy, intuition, commitment, navigating politics and long conversations (Thompson et al. 2017; Turner et al. 2017; Vereijssen et al. 2017; Fielke et al. 2018; Robson-Williams et al. 2018; Duncan et al. forthcoming). Importantly, the best people and approaches for achieving excellence might not be the same as those for achieving impact.

As a result, the practices required to meaningfully connect science with other knowledge producers and users for tangible outcomes (i.e. impact) can be far more complex, active and precarious than is portrayed in popularised accounts of linking knowledge to action, and of brokering and packaging excellent science (e.g. Bennett and Jessani 2011; Pielke 2007). It is in these real-world contexts that science can have impact, and it is in these contexts that governments are calling for researchers and scientific institutions to do just that (MBIE 2015, 2017, 2019a, 2019b). Important questions remain, however, about how this is to be done.

To address the how question, this paper presents the TDR Outcome Spaces Framework (OSF) as a framework for planning for multiple outcomes and assessing potential for research impact. We tested innovations to OSF, which we refer to as OSF+, with four of New Zealand’s National Science Challenges (NSCs).

Mission-led research in New Zealand: the NSCs

In 2013, 11 mission-led and outcome-driven NSCs were established by the New Zealand Government’s Ministry of Business, Innovation and Employment (MBIE) to address the country’s biggest and most challenging issues. The NSCs focus on issues identified through public engagement as having importance to New Zealanders (e.g. health care, nutrition, biodiversity, childcare, housing, natural hazards, climate change and natural resource management). Funding was approved for 10 years between 2014 and 2024, with a mid-term review in 2018, which approved funding for all NSCs to continue through to 2024.

A shift from science literacy to transdisciplinarity to achieve impact

The language used by MBIE to describe how the NSCs are expected to operate has changed from a largely multidisciplinary approach – with collaboration expected to occur among scientists and science institutions, and references merely to engagement with the public, stakeholders and end users in earlier documentation (MBIE 2015) – towards more inter- and transdisciplinary language that conceives of stakeholders, end users and Māori partners as research collaborators. For example, a 2015 performance outcomes logic framework requires an ‘increase [in] public engagement in science to lift STEM [science technology engineering and mathematics] skills and improve decision-making by the public’ (MBIE 2015, p. 10). By 2019 the requirements go beyond engagement and improving the public’s science literacy to include increasing the ‘impact and value’ of science and the science system by including ‘inter/trans-disciplinary research’ (MBIE 2019b, np).

Furthermore, while absent in 2015, the requirements now include ‘actively co-design (at the outset) and co-develop/create (along the way) research with stakeholders, end-users and Māori partners at all levels maintaining strategic focus on national scale issues’ (MBIE 2019b, np). In MBIE (2015, p. 10), activities included ‘develop and maintain the right capabilities and emerging talent’, but the 2019 version replaces ‘right’ research with ‘inter- and transdisciplinary research’ (MBIE 2019b). There are clearly expectations for research to be co-designed with relevant stakeholders, Māori partners, policymakers and communities, and this has been implemented in varying degrees across the NSCs (MBIE 2018). The significant shift in language has had important implications for the various NSCs and how they have undertaken their work and the challenges they have faced.

With a mandate to do mission-led science and deliver impact, the NSCs have become a platform for stakeholders, Māori partners and researchers to co-design research. As such, they provide an opportunity to evaluate the utility of OSF+ for assessing research impact potential and are a window into how New Zealand’s science system is accommodating TDR and the prospects for impact.

Introducing the OSF and testing two innovations (OSF+) for assessing research impact potential with NSCs

The OSF, as originally formulated by Mitchell et al. (2015), was identified by the first and second authors as a useful tool for discussing the how impact is being created within and across the NSCs. Involving stakeholders, end users and Māori partners as research collaborators inevitably broadens and multiplies the outcomes desired from research, yet these often not recognised or discussed, especially if the measures of success are scientific publications. A key feature of OSF is that it begins at the end (Mitchell et al. 2015). As such, it encourages research teams to discuss, agree and plan at the outset for multiple outcomes that aim to deliberately bring about change and promote sustainability (Mitchell et al. 2015, p. 86; see also Thompson et al. 2017). In particular, OSF was seen as useful for helping research teams have critical conversations with Māori partners about their values, visions and expectations, and to fulfil Vision Mātauranga commitments required by New Zealand’s science funding system.

Mitchell et al. (2015) identify three ‘transdisciplinary outcome spaces’ that are viewed as foundational for successful TDR: (1) a change in the situation; (2) contributing to knowledge stocks and flows; and (3) mutual and transformational learning by researchers and research participants. It is the co-development and negotiation of these outcomes with scientists, communities, stakeholders, research partners and policymakers that underpin and enable TDR research impact.

Key elements of OSF

Outcomes for bringing about a change in a situation might take the form of new policy or guidelines, or more encompassing or new criteria for assessing the merit of research or development projects. Outcomes in this space can be seen as levers that can be created through TDR processes to identify and catalyse changes. Outside-science research collaborators can help identify the need for these ‘levers’ as well as help a research team understand their opportunities and barriers. Indeed, the involvement of outside-science participants can be contingent on working to create or reform these levers, which is why it is important to identify and plan for multiple outcomes from the start.

The OSF places importance on relevant stocks and flows of knowledge, suggesting the need to develop rigorous but also accessible scholarly outputs and other forms of knowledge in order to make insights meaningful to research participants and the beneficiaries or end-users of research. Hence, in addition to academic articles in peer-reviewed journals, these might be in the form of decision-making tools, industry reports and interactive websites.

Mutual and transformational learning is also needed for both researchers and research participants, because such learning has the potential to increase the likelihood of persistent change. This requires creating or curating opportunities for learning experiences throughout TDR projects and afterwards.

Identifying at the outset what outcomes are desired, preferred or promised prompts a research team to think and plan upfront what might need to be done to deliver those outcomes in a project, and what is desirable for each of the various collaborators. An important feature of the OSF is that it is useful for visualising, negotiating and deciding where resources could or should be dedicated to deliver the outcomes within the three domains (Figure 1).

Figure 1. Conceptual map of the three outcome spaces (1. situation, 2. knowledge and 3. learning) indicating a transdisciplinary project within our field of vision that is itself embedded in the broader landscape (Source: Mitchell et al. 2015, p. 91 with permission).

Figure 1 shows an ideal profile with the three outcome spaces the same size. In reality, the size of the outcome spaces is unlikely to be equal, as there are always limited resources and there will have to be trade-offs in planning for multiple outcomes. There will also be a range of factors beyond resourcing that have the potential to influence the extent to which certain outcomes are achieved and how they relate to each other. Influences might include government policy, the epistemological commitments of researchers about how knowledge is or should be produced, ontological commitments about what exists in the world, and theories of change embodied within funding institutions and the minds of researchers. These factors and issues have important implications for how TDR is designed, undertaken and evaluated (Mitchell et al. 2015).

The project boundary acknowledges that resources are limited, as is the scope of the research. Hence, strategic decisions are required to plan for multiple outcomes in each of the outcome spaces. The project boundary also signals that outcomes can be planned to occur within and/or beyond the project. The field of vision is ‘limited to the experiences, knowledge and worldviews of the research team’ (Mitchell et al. 2015, p. 90) and acknowledges there are unknowns and potential shocks beyond the field of vision (Figure 2).

Figure 2. Example of outcome spaces profile: representing differences in investment across outcome spaces (Source: Created by D. Fam).

Innovations to OSF = OSF+

From previous experience in TDR, the authors considered the following two innovations to OSF as potentially useful for facilitating discussion about the potential for research impact with the NSCs.

Innovation 1: Revising the outcome spaces to delineate knowledge stocks and flows

We proposed a separation of knowledge stocks and knowledge flows (Figure 3). Identifying the resources dedicated to knowledge stocks (i.e. doing research and its publication in academic journals) versus knowledge flows (i.e. making the research available beyond those journals to those who need or want to use it) and the relationship between them, were identified as useful indicators of potential for research impact. It was thought this distinction between doing and publishing the research and its flow into broader society would bring into view worldviews that influence decisions about where resources are to be dedicated (i.e. not only producing knowledge but also outcomes for making it meaningful and useful). OSF with this innovation will be referred to as OSF+.

Figure 3. OSF+ with revised outcomes spaces framework delineating knowledge stocks from knowledge flows (Source: Created by D. Fam).

Innovation 2: Trialling a method of identifying intended, actual and desired outcomes

Visually representing the size and configuration of the outcome spaces (e.g. the degree of investment of a range of resources in each outcome space) were called ‘profiles’ and they were identified by the authors as a way to help the NSCs reflect on their mission-led mandates and potential to create impact across stages of their programmes. Given the mid-point review timing for the NSCs, we wanted to compare the outcomes intended by the NSCs when they started with what they actually delivered by the mid-point, and also what was desired for the second round of funding, having reflected on what had occurred in the first round of funding. To gauge potential impact, we hypothesised it would be useful for workshop participants to see how the OSF+ profiles changed across these three phases of their programmes. OSF+ appeared to be a promising tool that would allow the NSCs to reflect on their mission-led research and potential for impact, which we tested with researchers from four NSCs using the following methods.

Methods

Our research involved a collaboration between the authors to develop OSF+ and a one-day workshop. NSC researchers were invited by email to attend the workshop. The workshop was attended by eight researchers from four NSCs (two from two NSCs, one from one NSC, and three from the other NSC). Given the one-day format, we asked participants to work at a high level and at a whole-of-NSC scale. The workshop took place in February 2019 and was jointly facilitated by the authors.

Social ethics approval was obtained from the Manaaki Whenua – Landcare Research social ethics committee. Through the ethics process, concerns were raised about the anonymity of participants and professional risk for participants that might arise from comparing NSCs. To encourage participation and allow open discussion, commitments were given to participants that these issues would be fully considered if publishable results came from the workshop, and that they would be provided with a copy of any draft paper prior to publication. We believe the workshop has elicited important insights for the NSCs, policymakers and MBIE. While anonymity cannot be guaranteed, to address professional risk concerns and provide anonymity to researchers while making the findings publicly available, we have not named the NSCs, nor the researchers who participated in the workshop. To further provide anonymity, we have not referred to the roles of participants in their respective NSCs.

In line with the objectives of this paper, the purpose of the workshop was to:

1. Contribute to TDR theory and practice by testing OSF+ with New Zealand’s NSCs given their mission-led research programmes with inter- and transdisciplinary mandates;

2. Use OSF+ to examine how the NSCs are seeking to create impact, the barriers they have faced in delivering their inter- and transdisciplinary mandates and what potential there is for delivering the research impact expected from mission-led research

Participants worked with their NSC group to complete three OSF+ templates, which prompted participants to reflect on intended, actual and desired outcomes from their NSC. Figure 4 is an example of the working template.

Figure 4. Example of template used at workshop by participants to map outcome spaces of their respective Challenge (Source: Created by D. Fam).

Working through a step-by-step programme the authors developed prior to the workshop, participants were asked to consider the original intended outcomes when the NSCs were formed and categorise the intended outcomes into the four outcome spaces. They drew a ‘profile’ representing the level of investment and effort intended to deliver these outcomes. The participants then were asked to consider the actual outcomes they had achieved and invested in at the mid-term point, and draw a profile for the actual outcomes. Through comparing and contrasting how and why the outcome spaces differed between what was intended and what actually happened, participants completed a final profile envisaging how they would need to operate in the future and the desired outcomes necessary to meet the NSC missions. A final step of the workshop was to ask participants to compare their actual and desired profiles, and to identify barriers, risks and opportunities in moving from the former to the latter.

The outputs of the workshop are a set of three profiles representing intended, actual and desired outcomes for the four NSCs with broad descriptions of what the outcomes were. Another output is reflections from participants on perceived barriers, opportunities and risks, as well as notes taken on the day. Further outputs include an anonymous evaluation that asked participants to identify what aspects of OSF+ and the workshop activities they found useful. Data analysis on the benefits of using OSF+, and the reflections on the challenges of achieving impact involved an inductive and deductive approach: workshop discussions were synthesised into key themes (Merriam and Tisdell 2015) and worked templates were translated into graphics, with their interpretation guided by the science policy, TDR and OSF literature cited and discussed above.

Findings from workshop

Evaluating OSF+ to inform theory and practice

The separation of stocks from flows (innovation 1) helped identify different ways of working across the NSCs. For example, for Challenge A, small outcomes spaces for knowledge stocks alongside larger spaces for knowledge flows highlighted an initial intention of a reduced focus on creating stocks of knowledge on the basis that much of the science was already done; what was needed were connections across institutions and stakeholders to utilise existing knowledge (Figure 5). On this basis, OSF+ elicited multiple profiles for outcomes of mission-led research that shows its utility for assessing research impact and TDR capacity (compare Figures 5 and 6).

Figure 5. Challenge A perceived outcomes: Intended (left) and Actual (right) outcome spaces. Knowledge broker profile and a catalyst theory of change.

Participants noted that separating knowledge stocks from flows was a useful indicator of both research impact and societal impact. Although our participants had not worked with the OSF in its original format, it is clear from the profiles that this separation helped participants distinguish between the conduct of research outcomes versus the flow of knowledge outcomes. For example, Challenge B realised that, notwithstanding its intentions, it was putting most of its resources into building knowledge stocks, with limited attention to making knowledge flows accessible to stakeholders and partners (Figure 6). Discussion of the relationship between knowledge stocks and flows and how accessible the flows were revealed Challenge B was focused on getting papers published in high-quality journals, which is an outcome required under MBIE’s performance outcomes framework. It was also explained there was an expectation that the number of papers match the amount of funding.

Figure 6. Challenge B perceived outcomes: Intended (left) and Actual (right) outcome spaces. Knowledge deficit theory of change.

We hypothesised that the OSF+ profiles would be useful for helping the NSCs compare what was planned with what was done (Innovation 2). Participants found this exercise of mapping intended (i.e. what the NSC set out to do), actual (i.e. what the NSC ended up doing) and desired (i.e. what it will do better in the future) outcomes to be powerful. Participants noted this staged mapping process allowed them to explore barriers, trade-offs, constraints and risks to examine why actual outcomes did not reflect intended outcomes, and to identify the potential opportunities for doing things differently in the future.

Using OSF+ with NSCs to assess research impact potential

As well as testing the above innovations, several insights from applying OSF+ emerged from the workshop. Given the large-scale, systemic and often slow-moving nature of the issues the NSCs seek to address, the project boundary was interpreted temporally and highlighted the need for NSCs to plan for a legacy beyond the life of their programmes. Indeed, we found that where participants placed their outcomes spaces in relation to the project boundary revealed quite different theories of change among the participants.

For example, as discussed, Challenge A recognised at the outset that much research already existed to address its objective. Consequently, it had been quite deliberate in dedicating resources to utilise existing knowledge and connect it with knowledge users (i.e. knowledge flows) rather than produce new knowledge (i.e. knowledge stocks). For the new knowledge Challenge A identified was needed, it was using seed funding to encourage others to undertake research to leverage more research from its limited resources. This meant the knowledge stocks and flows were placed just within the project boundary, with the largest proportion of the outcome spaces sitting outside the project boundary (Figure 5).

This placement represented a legacy that was intended to be achieved outside the NSC, and resources within the project were intended to deliver legacy outcomes. The project boundary delineated outcomes inside and outside the NSC, with the legacy expected to be delivered by others empowered or enlisted by the NSC. In other words, outcomes and the NSC legacy were planned to be achieved through catalysing and facilitating research. Hence, this NSC’s theory of change is reflected in the OSF+ profile as a catalyst and a knowledge broker.

Challenge A (Figure 5) contrasts with Challenge B (Figure 6), whose participants recognised their NSC embodies a knowledge deficit theory of change (i.e. knowledge stocks are assumed accessible through academic publications and will reach imagined end users eventually or become sought after when end users recognise they need to change their behaviour).

Figure 6 shows that although the NSC intended to have a legacy and had envisaged outcomes for knowledge stocks and flows (left), what occurred was that the push for ‘high-quality’ science to meet other MBIE performance criteria resulted in the NSC dedicating significant resources and attention to knowledge stocks and relying substantially on academic knowledge flows. The limited flow of knowledge is reflected in its relatively small outcome space and placement within the project boundary (right). This NSC’s legacy was limited to knowledge stocks, with the placement of this outcome space beyond the project boundary reflecting an expectation the stocks (i.e. academic publications) would endure (and continue to be available) beyond the NSC and eventually deliver uptake and, thereby, impact.

The project boundary prompted Challenge C to place its intended change in the situation outcome space outside the project boundary (but within the field of vision) (Figure 7 left), on the basis of an expectation that the impact of the science work was unlikely to make its way to end users until after 2024.

Figure 7. Challenge C perceived outcomes: Intended (left) and Actual (right) outcome spaces. Active engagement theory of change.

However, what occurred for this NSC was that knowledge flows were achieved with resources and personnel dedicated to engagement with the public, policymakers and other stakeholders identified as potential users of the NSC’s work. Activities undertaken and materials produced to make knowledge flow were seen by Challenge C participants as an outstanding outcome, which is reflected in the relative sizes of the knowledge stocks and flows outcome spaces Figure 7 (right). This success is also reflected in the placement of the change in the situation partly within the project boundary for actual outcomes (Figure 7 right). In other words, this NSC envisaged it would not have an impact in the short term and would have to rely on its legacy, but, on reflection, it was concluded that the NSC did have impact and researchers were able to use OSF+ to articulate how and why.

Furthermore, the changes in the project boundary and field of vision (Figure 7 right) reflect how political imperatives and changes in government policy to respond to urgent issues reshaped the project boundary (in this case reducing the NSC remit) as well as expanding the field of vision. It was noted that the latter did not come with resources to extend the project boundary to address the issues that had come into the NSC’s field of vision.

Similarly, Challenge D (Figure 8, left) participants explained how a change in government direction had so fundamentally changed expectations for the NSC’s outputs and outcomes, the field of vision was described as having been reduced to inside the NSC project boundary. This was the result of requirements for peer-reviewed knowledge stocks for a significant period, which had negative implications for the broad range of collaborators who had been involved with the NSC to deliver impact through co-design activities. Participants explained that complaints from stakeholders to those governing the NSC resolved these issues. Hence, the co-design and collaborative research this NSC was doing was able to continue. The large learning outcome space highlights the embedding of social scientists within research teams and their familiarity with co-design and collaborative research challenges and opportunities.

Figure 8. Challenge D perceived outcomes: Intended (left) and Actual (right) outcome spaces. Collaboration theory of change.

The workshop revealed that interpreting the project boundary temporally was useful for thinking about the legacy an NSC might have, the outcomes needed to deliver that legacy, and how the outcome spaces needed to intersect in order to do so.

The unexpected: a lack of recognition for mutual and transformational learning

While the profiles show learning as a relatively large area of investment for most NSCs, the outcomes included building internal structures and capacity for Challenge A, and building expertise and capacity to generate knowledge stocks for Challenge C. Challenges B and D had delivered initiatives to spark transformation and establish co-design, but these were undermined by imperatives to build knowledge stocks with assumed flows through the production of peer-reviewed publications in high-quality journals. Overall, working with this outcome space highlighted for participants that mutual and transformational learning was more incidental than purposeful, and was a blind spot that represented opportunities in the future. These discussions also raised questions for the research team about how participants conceived learning and whether they were thinking in terms of transformational and what that might actually involve.

The unexpected: a lack of specifics for a change in the situation

The ‘change in the situation’ outcome space was interpreted by several participants as the size of the problem the NSC was required to address, which was recognised as enormous – if not overwhelming – and getting larger (Figures 5, 6 and 8), or so distant it was difficult to specify how the NSC would or could change the situation (Figure 7). In other words, the situation outcome space was interpreted as the impact the NSC needed to have into the future rather than outcomes purposely designed and undertaken to make change. While this interpretation could have been due to the short period of time available to work with the framework, participants’ responses provided important insights. For example, it was explained by some participants that talking about outputs rather than outcomes was more realistic and helpful for scientists working on small parts of larger pieces of work. Notably, outputs as knowledge stocks were deemed to be more tangible and controllable given how difficult it was to envisage how or when change in the situation would occur given the time lags between action and results. This exchange highlighted the difficulty for researchers working with aspirational goals that are largely unspecified in practical terms.

OSF+ as a useful evaluation tool?

Participants were asked how useful they thought OSF+ could be for discussion, planning, reflecting on and potentially evaluating the impact of their NSC. All participants were very positive about OSF+. Features noted were:

• its alignment with the MBIE performance framework

• the way OSF+ forces thinking about what ‘transdisciplinary’ means and how it could be applied in practice

• the way it focuses attention on where effort should, or could, be directed

• how effort contributes to overall TDR outcomes

• the way it makes explicit decisions on resources and trade-offs

• how it can reveal implicit theories of change that exist behind where effort is placed across the outcome spaces.

A participant noted that ‘OSF+ was useful for thinking about the different facets that are needed to achieve outcomes and how these might be balanced’.

Participants were also asked what aspects of OSF+ were useful. The outcome spaces were identified as a ‘heuristic’ for discussion within research teams. Also useful was the mapping exercise with the intended, actual and desired outcomes, and the different profiles this generated for comparison. It was noted these aspects revealed a stark contrast between ‘talk and walk’ (i.e. talking about co-design and TDR and actually doing it), how context affects project scope and field of vision, and how learning is an area that had not been adequately considered and where effort is needed. A participant noted, ‘It helped me think of reasons why there may have been difficulties in the past. It helped me understand where we might need to focus our future efforts more to achieve outcomes (e.g. learning).’

Participants were also asked what aspects they found challenging. While most saw the framework as straightforward, a participant who was already thinking about how to put it into practice noted the effort required in ‘putting sufficient detail into the framework to make it helpful as a tool for prioritising or making decisions about the timing and effort required to be “successful” (the achieving an impact)’.

To identify what participants thought might be missing, they were also asked what other outcomes should be included in the framework. Several participants identified that short-, medium- and long-term outcomes could be articulated and planned for (which would align with the MBIE performance framework). It was also noted that learning could be refined to differentiate internal and external learning within that outcome space. Another participant mentioned the project boundary and field of vision, and the importance of thinking about how it can expand and contract. A participant also suggested knowledge flows should be conceived as multi-directional (i.e. not only from but also to researchers).

Discussion

Evaluating OSF+ to inform theory and practice

We hypothesised that separating knowledge stocks and knowledge flows could be a useful way to assess research impact and TDR capacity within the NSCs and New Zealand’s science system. We also hypothesised that this separation could identify implicit theories of change (Oberlack et al. 2019) that underpin decisions about where resources are dedicated and their likelihood to deliver impact. To test OSF+, we asked participants to estimate where their NSCs were dedicating resources to achieve what outcomes, and how this differed between what was intended and what happened. This was a powerful and enlightening exercise for the authors and participants. From the workshop evaluation, the key to this was that participants could visualise how the OSF+ profiles changed across the different scenarios, and the staged process made the tensions, trade-offs, barriers and opportunities visible. The exercise highlighted the importance of reflecting on the relationship between knowledge stocks and flows, what kind of flows the NSCs were focused on (e.g. academic publications or materials useable by broader society), and what else might be required for impact (e.g. curated transformative learning experiences or policy interventions informed by practitioner knowledge and experience).

With outcomes to deliver a change in the situation unspecified, and participants conceding that limited attention and resources had been dedicated to transformational learning outcomes, the NSCs have been predominantly working in two outcomes spaces; knowledge stocks and knowledge flows but with the greater attention on knowledge stocks. Arguably, this focus on knowledge stocks is a legacy of the knowledge deficit paradigm embedded within the science system, whereby a change in the situation is deemed to take care of itself through the eventual diffusion of academic outputs. These ‘institutional logics’ (Turner et al. 2016, p. 99) have important implications for TDR and practice, and the potential for delivering impact if the measure of success across the science system remains publication in academic journals, which several participants maintained they were directed to focus on notwithstanding requirements for co-design and to do TDR.

We found that the concept of knowledge flow, in contrast to funder language of knowledge uptake, better conveys the need for researchers and research institutions to be more proactive in facilitating the flow of knowledge rather than waiting for knowledge stocks to be somehow linked to end users, or found by them, which the notion of ‘uptake’ implies. Thinking about knowledge flow as multi-directional, as suggested by a workshop participant, is an important contribution to OSF+ theory and TDR practice.

Using OSF+ to assess research impact potential

The innovations to OSF (i.e. OSF+) showed that the relationship between knowledge stocks and knowledge flows differed between the participating NSCs, with each broadly (although not exclusively) reflecting profiles that could be identified with different theories of change (Table 1).

Table 1. Theories of change that align with participating challenges.

Challenge	Theory of change
A	Catalyst – change occurs through intervening and connecting existing knowledge and knowledge producers with knowledge users
B	Deficit – change occurs through creating knowledge stocks to fill knowledge gaps identified by the NSC and eventual uptake
C	Engagement – change occurs through identifying knowledge users and tailoring materials and activities to their needs
D	Collaboration – change occurs through partnerships with stakeholders to collaboratively identify knowledge gaps and issues to produce knowledge stocks and flows together

These different theories of change reflect different philosophies and capabilities across the NSCs and how time, resources and decisions have been dedicated to different outcomes. While each participating NSC intended and achieved knowledge flow outcomes in varying and successful ways, publications in academic journals dominated the outcomes for all NSCs for a range of justifiable reasons (e.g. fulfilling contractual requirements, meeting multiple research and personal performance measures, achieving tangible and controllable outputs). Crucially, the extent to which stakeholders, Māori partners and knowledge users can be collaborators in knowledge production appears highly vulnerable, with efforts often thwarted and networks undermined by what are mixed signals and conflicting measures of success and required outputs.

These findings raise important questions about New Zealand’s science system and its ability to accommodate TDR and deliver societal as well as academic impact (i.e. research impact). For example, requirements for science excellence encourage and validate the production of purely academic knowledge stocks and flows. As discussed above, these outputs substantiate science excellence beyond the NSCs (e.g. the Endeavour Fund and Marsden). However, they are appearing incompatible with TDR to deliver impact with research collaborators who are likely to see the production of scientific papers as irrelevant if not obsolete. For example, Challenge B was identified as underpinned by the knowledge deficit theory of change. Challenge B’s profile indicates that this theory of change, which aligns with that of the current science system (see Turner et al. 2016), has the potential for the least societal impact. It appears that societal impact and TDR are being treated as add-ons to the existing science excellence system, when what is required is a reimagining of how science is done and how it is governed (Wyborn et al. 2019).

Conclusions

Evaluating OSF+ to inform theory and practice

OSF and our innovations (OSF+) have been tested in New Zealand with a selection of researchers working with its mission-led impact-focused NSCs. It was found that OSF+ has considerable potential as an evaluation tool for assessing impact potential, especially by comparing intended, actual and desired outcomes. Being able to visualise a sequence of changing outcome spaces profiles that illustrated where efforts and resources were expected to be directed to plan for and deliver multiple outcomes, where they ended up, and where they needed to go in the future, was identified as a particularly powerful feature. Separating knowledge stocks from knowledge flows, OSF+ was identified as being able to reveal implicit theories of change, which was also recognised as a compelling feature. Conceiving knowledge flows as multidirectional (i.e. from a TDR research team to stakeholders and end-users and vice versa) is an important contribution to OSF+.

Using OSF+ to assess research impact potential: policy implications

This research opens important questions about New Zealand’s science system that has been built on a knowledge deficit theory of change. Our workshop participants identified several barriers and risks in doing mission-led research, and tensions in achieving science excellence alongside impact within a system that rewards narrowly defined excellence. While expectations for doing and building TDR capacity are necessary, they are clearly not sufficient. Importantly, in light of MBIE’s (2019a) recent The Impact of Research: Position Paper, these expectations now appear to extend beyond the NSCs. Starting out as endeavours within and across scientific institutions and with impact conceived largely in terms of science literacy, TDR and what that requires was not envisaged when the NSCs, their missions, structures, timelines and funding were established. Doing TDR and what is actually required to create impact is not yet fully understood or appreciated. It takes considerable time, funding and special skills. With values and interests at stake, it is challenging to do and can be extremely uncomfortable for researchers, stakeholders and research partners. TDR also raises challenging questions about what is knowledge and whose knowledge counts? These questions (and the answers) are especially important in New Zealand for tangata whenua. Creating impact through TDR is vital but challenging. Measuring impact is important but it does not address the how question or recognise how the system needs to change to accommodate what TDR actually requires. This research highlights the value of the NSCs as experiments in attempting TDR and research focused on impact. Working with researchers within the NSCs to gather deeper insights on the issues raised in this paper will be important for going forward to address the how question and to build a fit for purpose science system that incentivises TDR and fosters a mutually supportive relationship between research impact and science excellence.

Disclosure statement

No potential conflict of interest was reported by the authors.