Three frameworks to understand and manage social processes for integrated catchment management

Abstract

Integrated catchment management (ICM) initiatives involve many complex social interactions. Project leaders and participants face challenges in managing multiple demands for engagement, communication and integration of different knowledge across agencies, sectors, research disciplines and communities. Social frameworks can be practical management tools that help project leaders and participants: (1) make sense of the social and management context of a project; (2) design strategies to meet social process needs such as communication and engagement; and (3) evaluate the effectiveness of the project with a view to improving it. This paper examines the role of social frameworks in supporting ICM research in the Motueka catchment over 10 years. It reviews use of the ISKM (Integrated Systems for Knowledge Management) framework for sharing information between different stakeholder groups and the Orders of Outcomes framework for evaluating outcomes over long periods. In particular, it introduces the Social Spaces framework as a new tool for visualising diverse communication and collaboration needs across a project. We conclude with suggestions on using frameworks in conjunction with participatory evaluation to build capacity and strengthen relationships among project participants.

Keywords:

• integrated catchment management
• frameworks
• evaluation
• programme development
• social processes

Challenges of integration and the role of frameworks

Integrated catchment management (ICM) is a strategic approach to land and water management designed to help multiple stakeholders make informed decisions and take co-ordinated action to manage a complex environmental system (Mitchell & Hollick 1993). It uses a systems approach to understand interactions between physical and social elements that impact on the management of natural resources within a catchment area (Allen et al. 1998). This requires new research frameworks to link different bodies of knowledge (Falkenmark 2004). It also requires novel avenues for stakeholders such as agencies, sector groups, landowners, tangata whenua (indigenous peoples) and non-governmental organisations to acquire new information, and interpret and apply it to their own context. As such, the challenge for an ICM project is as much about building collective understanding of a complex situation, and enabling science information to be useful in decision-making, as it is about researching to understand the biological and physical environment (Margerum 1999; Allen & Kilvington 2002).

The practice of running an ICM initiative goes beyond the already significant task of enabling cross-disciplinary collaboration, to managing an array of social processes, such as public participation and engagement, multi-stakeholder inquiry, and conflict management. Klein (2004) in Lélé & Norgaard 2005) describes twin challenges of integration: horizontal integration (across disciplines) and vertical integration (across experts, policymakers and community). This represents unfamiliar territory for many leaders and participants in ICM initiatives. With whom, when and where should they be concentrating efforts to stimulate dialogue and collective learning? Does this change over time and if so how? Answers to these questions have important resource implications for any project and require active design and management of the social system of the ICM initiative itself.

Just as biophysical models enable visualisation, interpretation and testing of hypotheses about biophysical systems, social frameworks that describe and unravel complex interactions in ICM can be useful tools to help project leaders and participants understand the social system in which they operate. They can be used to introduce new relevant social theories to the integrated catchment initiative, as well as prompt questions and stimulate discussions on how to make progress. Frameworks can also be the basis to an evaluation process designed to review programme achievements. We propose that frameworks can be useful to ICM initiatives in three ways: (1) to make sense of the social and management context of a project; (2) for designing strategies to meet social process needs such as communication across many groups; and (3) as the basis for evaluating the effectiveness of the project with a view to improving it.

The next sections of this paper outline three frameworks and their use within ICM Motueka research. ICM Motueka research was a 10-year multidisciplinary research initiative based in the South Island of New Zealand. It was designed to improve the management of land, freshwater and coastal environments in catchments with interacting, and potentially conflicting, land uses. Multiple research and resource management agencies have been involved in the programme. Its distinction as a research programme was its intention not only to provide research information to catchment management agencies, but also to influence the integrated nature of management. For detail about the ICM programme, see Fenemor et al. (2011).

This section begins with an outline of the three frameworks and then discusses their use within ICM Motueka research. The paper concludes with the potential of linking frameworks with participatory development evaluation to help participants across the project as a whole grapple with the challenges of integration.

Use of frameworks in ICM Motueka research

Over 10 years, the ICM Motueka research programme explored the value and content of frameworks for successful ICM application. It used different social frameworks to generate ideas and develop strategies. Significant among these were the ISKM (Integrated Systems for Knowledge Management) framework (Allen & Kilvington 2002; Allen & Jacobson 2009), the Orders of Outcomes model (Olsen 2003), and the Social Spaces framework outlined in this paper and first discussed in Kilvington & Allen (2007).

These frameworks each offer something different to the challenge of managing long-term multi-stakeholder ICM initiatives. ISKM is a framework for understanding the different steps required in a long-term-engagement approach to adaptive management in catchments and other integrated resource management fields; Orders of Outcomes (Olsen 2003) is a generic logic model for ICM that assists planning for the different levels of outcome that can be expected in projects over time. The Social Spaces framework visualises how the diverse communication and information exchange that take place simultaneously between different groups in a complex ICM initiative can contribute towards successful ICM outcomes.

ISKM framework

The ISKM was first developed during a long-term, multidisciplinary research programme in the highly contested and often polarised area of high country management in the South Island of New Zealand (Bosch et al. 1996; Allen et al. 1998). The focus of the ISKM framework (Fig. 1) is to provide an organised set of principles that guide engagement in real-world problem situations. It can be regarded as a project management process for developing and sharing information when participation of multiple actors and a situation of high complexity demand a focus on systems thinking, collaborative decision-making and experiential learning.

Figure 1  Key phases of Integrated Systems for Knowledge Management (ISKM; Allen & Kilvington 2002).

The ISKM framework is designed around the steps of adaptive management. Two phases are involved: the first supports finding out about a situation and the second aims to take action to improve the situation. As Allen and Jacobson (2009) explained, activities associated with the first phase (1) establish a climate for change with the different parties involved; (2) set goals and objectives (including joint problem framing); (3) search for information; (4) develop a shared understanding; and (5) create action plans to address the issue at hand. Monitoring plans also need to be developed to track progress and help ensure that the action plans remain valid and on course. The final activity in this first phase of ISKM involves the development of a management information system for the benefit of the wider community of stakeholders. Computer technology is often relevant at this stage as it offers ways of organising information that is easily accessible to a range of audiences.

The second phase of ISKM stresses the need to develop feedback loops or pathways for using the subsequent monitoring and evaluation information to generate new problem definitions and set next-stage plans. This is essential to support a collaborative-learning, self-improving environment.

Orders of Outcomes framework

Developing the range of social, biophysical and institutional outcomes that support evidence of good policy and practice in complex social and environmental situations is challenging, not least because results in these areas can take some years to materialise (Allen & Apgar 2007). Accordingly, it is good to visualise how these outcomes can form a logical sequence over long time-periods. One approach for grouping the outcomes of an integrated governance initiative is known as the Orders of Outcomes model (Olsen 2003; UNEP/GPA 2006). It highlights the importance of changes in state (such as better environmental or social outcomes). Furthermore, it recognises that later changes in state (end-outcomes) rely on earlier changes in the behaviour of institutions and key stakeholder groups (intermediate outcomes). In this way, the model h elps ICM project leaders and participants plan activities in succession so these build on each other over time (Fig. 2). The model also reminds us that changes will show up differently at different scales.

Figure 2  Orders of Outcomes model approach to monitoring and evaluation (adapted from Olsen 2003).

The Orders of Outcomes model defines differing levels of outcomes and their expected logical sequence for achievement in complex situations such as ICM (Fig. 2). It helps managers plan activities so these build on each other over time. Olsen's (2003) model describes 1st Order outcomes as the enabling conditions required for sustainable ecosystem management. These include establishing core relationships and formal commitments. The 2nd Order outcomes are those changes in behaviour of relevant institutions and groups affecting the ecosystem. These changes include the redirection of funding towards sustained adaptive management processes.

The 3rd Order outcomes are termed the harvest and mark the achievement of specific societal and environmental quality goals that are central to the ICM initiative, such as improved water quality or equitable water allocation. The 4th Order outcome is referred to as sustainable management and centres on the establishment of a management approach that can maintain equilibrium between environmental and societal goals. Often regarded as an unattainable ideal, this is nevertheless the long-term goal of ICM and steps towards this need to be rendered recognisable and measurable.

Social Spaces framework

The Social Spaces framework is a tool for interpreting and reflecting on processes of communication, dialogue and learning within an ICM initiative. It offers a typology of critical social networks and relationships, which can be used to map current communication and collaboration efforts, and plan future efforts. Unlike the previous frameworks it was derived directly within ICM Motueka research itself, so is described here in more detail.

Engaging with constituent communities is a key component of all ICM initiatives. This engagement has several purposes, including better interactions between researchers and developers of technical information from different disciplines, and across different institutions; enhanced contribution of science information to policy development; and improved links among stakeholder communities leading to co-ordinated management responses.

ICM Motueka research needed to assess the effectiveness of current engagement efforts at developing a good social learning environment. In 2006, an evaluation into community engagement in the project took place using semi-structured interviews with stakeholders and research collaborators, including researchers from different institutions, members of local management agencies and community members who took part in various programme events (Kilvington & Allen 2007). Interviewees were asked to comment on interactions between participants in the programme, and their experience of the various programme engagement activities.

This review showed a wide range of activities, with multiple actors, were already happening in the programme. These included face-to-face meetings, newsletters, online networking sites, workshops, field trips and group meetings. It also exposed that, as is characteristic of a complex, adaptive system (Rittel & Webber 1973), it was unpredictable and to some seemed quite messy. With different projects at different stages, people frequently made new connections and took existing relationships in different directions. This richness of opportunities for networking is one of the primary engines for integration. However, it also leads to questions about how to respond to the many different opportunities for communicating within the programme, and how to know whether current efforts are building the social learning capacity of the programme. Finding some means by which the programme participants could assess the merit of the different avenues for engagement, and track progress, was clearly important.

From the review, a framework was derived that combined theory about important elements in the social system of an ICM initiative with the interviewees’ feedback on the current engagement activities in the programme. Termed the ‘Social Spaces framework’, this is a typology of different levels of communication and collaboration in the programme. These levels are analogous to what Price (2003) describes as the multiple social spaces within which the process of generating, debating and using science knowledge in the programme takes place. These social spaces comprise their own unique boundaries, their own narratives and their own contestations and negotiations. They are characterised by specific norms of engagement, core relationships and particular intentions.

The Social Spaces framework identified four virtual spaces of communication and collaboration (Fig. 3). Each of these represents domains of information exchange and knowledge development within the ICM programme and within the Motueka catchment, and are relevant for characterising any ICM project:

1.	Central research collaboration space or interdisciplinary space;
2.	Transdisciplinary space—where research connects with real-world problems;
3.	Information-exchange space—the intersection between the programme and the general audience of stakeholders and interested parties in the Motueka catchment;
4.	Intersection with the wider ICM community—the national and international linkages where the programme intersects with others working on ICM initiatives.

Figure 3  Social engagement spaces of the integrated catchment management (ICM) programme.

Space 1: The research collaboration (interdisciplinary) space

This space encompasses the relationships between the research partners from different disciplines and institutions. The goal of this space is to promote integrated work across disciplines and between institutions in order to build the research understanding of the catchment environment (Allen et al. 2011). The focus for the communication and interaction activities in this space therefore is to build a good collaboration environment. Important issues include: how research projects are organised, resource allocation, intellectual property agreements, establishing the roles and protocols of research, agreeing outputs and understanding working relationships. The challenge in this space is to create opportunities for different disciplines and institutions collectively to shape a research agenda and to grapple with different perspectives on catchment management research.

In the ICM Motueka research, a number of activities supported the communication and collaboration goals of this space. They included regular meetings between all researchers (not just programme and project leaders) particularly at the programme's annual general meeting. Also, the online workspace Confluens was designed to be a forum for discussion between participants from different institutions to debate and share ideas and resources.

Space 2: The transdisciplinary space

Space 2 is where research connects with real-world problems and may be regarded as the heart of any ICM initiative. It is here termed ‘the transdisciplinary space’ because its focus is on knowledge building around complex catchment management issues using the collective experiences and understandings of stakeholders from a range of technical and practitioner backgrounds (Allen et al. 2011). The goals of this space are to enable collaborative interpretation of both science- and non-science-generated information, and the development of ideas through negotiation. This requires the cultivation of opportunities for dialogue between technical experts, landowners, resource users, tangata whenua, sector groups and others.

Activities in this space need to be designed with an awareness of key factors inherent in good learning environments (Allen et al. 1998). These include: clearly identified issues around which there are diverse viewpoints, and which have bounded conflict; and discussion processes that investigate existing assumptions and foster ability to integrate new knowledge alongside existing ideas. Also important is capacity for systems thi nking, given the focus of inquiry is to develop management approaches responsive to the complexity of a catchment system.

Capacity for individuals and organisations to work in Space 2 takes time to develop. Successful interactions in this space depend on high levels of trust, strong networks and facilitated situations that encourage participants to work hard at processing information. A number of initiatives in ICM Motueka research were aimed not simply at meeting the needs of this space per se but as an investment in developing the abilities of programme participants to engage in transdisciplinary work successfully. These included the Community Reference Group—an informal forum that promoted new ways for researchers, resource users, and landowners to share knowledge (Allen et al. 2011).

Space 3: The information-exchange space

Space 3 is where the ICM initiative interacts with the wider catchment community, i.e. reaching beyond communication between programme participants and stakeholders actively involved in the project. The immediate goal of this space is to provide information in a range of forms for uptake by diverse audiences. Its aims are to increase awareness of ICM, to both generate and satisfy curiosity about catchment management issues and increase the programme's relevance to the local catchment audience. The challenge of this space is to create a range of opportunities for people to pick up new ideas that are appealing and responsive to the needs of different stakeholders. The success or otherwise of communication initiatives in this space can be influenced by existing networks and historical interactions between the core programme participants and the wider community.

Throughout its 10 years, ICM Motueka research put much effort into developing conduits for information dissemination and for promoting awareness of ICM at a local level. These included newsletters (e.g. Catchment Connections), a public website (icm.landcareresearch.co.nz), public participation opportunities at annual general meetings, a freely available CD Rom synthesising research findings, as well as field days and workshops.

Space 4: Intersection with the wider ICM community

This fourth space represents the links between an ICM initiative and the global and national community of researchers and managers involved in ICM. The development of communication and collaboration in this space is critical for both current and ongoing development of capacity in ICM. Networks in this space can be easier to develop than those in the other three spaces, as they involve communities of interest with common language and similar concerns. In contrast, Spaces 1–3 are primarily geographic communities (i.e. located in the Motueka catchment) and made up of members that have different ways of framing catchment management issues. ICM Motueka research had a number of active links into the wider ICM research and practice community including inviting international and national ICM researchers and practitioners to programme events, and participation in the UNESCO HELP (Hydrology for Environment, Life and Policy) programme (Bonell 2004).

Using frameworks to support ICM initiatives

Frameworks such as those outlined above are a useful way of introducing new relevant social theories to the ICM initiative. These can include ideas about how groups learn, how to structure participatory processes, new approaches to governance or group dynamics. They can be used by those planning and running an ICM initiative to prompt general questions such as ‘what does this framework suggest about ways to engage people on catchment management issues?’ In this way, frameworks become part of the overall conversations around programme design and implementation.

Overall, the three frameworks discussed here each met different needs for understanding and supporting the social process of ICM projects. They can be used throughout an ICM initiative but can be particularly valuable in early planning stages. Table 1 summarises the challenges and common questions that are met by the different frameworks, and indicates how these frameworks can be used in an ICM project management process. In the case of ICM Motueka research, the ISKM framework was adopted as the guiding premise for the overall programme approach, recognising that one of the main aims of the programme was to support a range of stakeholder groups in sharing and understanding information. Later in the programme's history, Olsen's (2003) Orders of Outcomes was introduced to clarify the wider pattern of outcomes emerging in the programme. The Social Spaces framework was specifically designed to meet a need for greater clarity around the range of communication and engagement activities emerging in the programme.

Table 1  Summary of uses for Integrated Systems for Knowledge Management (ISKM), Orders of Outcomes and Social Spaces.

Framework	Integrated catchment management challenges		Using the framework
Integrated systems for knowledge management framework (Allen 2001)	Managing information from different stakeholders in integrated and adaptive management initiatives	What steps do we need to go through to support a learning-based programme?	• Beginning of programme: overall framework guides conceptual structure of programme
			• Throughout programme: use to judge the need for further tools/approaches to support different steps (e.g. stakeholder analysis (Allen & Kilvington 2009) to support step 1 entry and contracting)
Orders of Outcome framework (Olsen 2003)	Understanding and predicting different outcomes and outputs for the initiative	Within what time frames can you expect different outcomes to be delivered?	• Beginning of programme: used to inform project planning discussion
		How will the project timeline intersect with regional planning and management processes?	• Beginning of programme and throughout: as basis to programme logic model evaluation exercises to assess progress and make changes
Social spaces framework	Understanding and planning for communication and dialogue across multiple audiences	What methods of dialogue, communication will be most effective throughout the project?	• Framework needs to be validated within project (do spaces correspond to reality?) May require programme survey/review
		How could these differ according to stakeholder needs and learning demands?	• Basis of a participatory evaluation exercise to review how existing communication strategies meet the needs of the different social spaces
			• Used to plan future communication strategies

This ongoing use of different frameworks is typical of complex long-term initiatives that require some planning at the outset but also need capacity to be responsive as situations change and new challenges emerge. However, one of the main questions facing ICM project leaders is how to structure discussion to involve other programme participants. One option is to use these frameworks as the basis for a participatory, developmental evaluation process designed to both review what the programme is achieving, and encourage thinking about creative new directions.

Linking frameworks to participatory evaluation

Using frameworks in evaluation means they become a new lens for project managers, participants and stakeholders to examine how the project's social system is functioning, and to critique and interpret the value of their current strategies and to make plans for future actions. In this way, they become part of a developmental process that builds capacity within the programme (Morgan 1999; Brickmayer & Weiss 2000). For example, using participatory processes in a developmental way, such as workshops that invite stakeholders from across the project, not only improves the shared understanding of the project among participants but also builds capacity within the project for dialogue and reflection (Fetterman 2002)—which are critical components of integrated environmental management (Keen et al. 2005). Importantly we are using the definition of evaluation that encompasses any structured process of reflection and consideration (rather than the more limited definition of evaluation that refers to assessment of merit and judgement). There are many ways to involve participants in developmental evaluations, and we discuss the merits of the three different approaches used here.

ISKM evaluation—the checklist approach

In 2006, the ISKM framework (Fig. 1) was used as a basis of an evaluation exercise in an aquifer research programme (Integrated Research into Aquifer Protection, IRAP) that was designing a model to predict nitrate leaching from different land uses to support resource management decision-making (http://www.irap.org.nz).

The IRAP programme and ICM Motueka research were both collaborations that operated at multiple levels, between researchers, between institutions, across disciplines and between the potential end-users of science and the science providers. As such they shared a common need for understanding, planning and maintaining these relationships, and moreover for advancing the development and effective use of new knowledge. In the interests of promoting a collective appreciation of how the different elements of the IRAP collaboration and tool development were progressing, a participatory evaluation process was designed based on the ISKM framework. A checklist of prompts and questions (Table 2) were used in a facilitated exercise held with the IRAP programme's End User Advisory Group. The overall purpose of the evaluation exercise was not to judge the programme but rather to enable programme participants to consider different aspects such as the interactions between the end-users and the researchers of the programme and raise issues they thought needed further work.

Table 2  Checklist based on Integrated Systems for Knowledge Management (ISKM) framework and sample questions.

1	Goals
	What are the main goals of the IRAP programme? What are the goals of the partnership organisations involved in the programme?
2	Entry and contracting
	Who is going to use the tools/models/decision support system from IRAP? Are those people involved?
3	Accessing relevant data, information, and knowledge
	Where is most information coming from to develop the IRAP models and is the balance of different sources appropriate?
4	Dialogue and negotiation
	What processes are there in IRAP for dialogue and negotiation around information and knowledge? What happens when there are divergent views?
5	Implementation and review
	Are you setting up ways to use monitoring information from management to validate/update the models?
6	Building the climate that makes it work
	How well aware are you of the key political and strategic relationships necessary to ensure the IRAP models are trusted? Are there effective mechanisms for communicating learning from IRAP to wider audiences?
IRAP, Integrated Research into Aquifer Protection.

The checklist evaluation began by identifying the goals of IRAP, and then covered four areas of the operation of an integrated research programme based on ISKM. These were:

•	Entry and contracting (who is and should be involved in the programme and how they are brought in);
•	Accessing relevant data, information, and knowledge (drawing together relevant information from different parties);
•	Dialogue and negotiation (processes of making sense of different participants’ contributions);
•	Implementation and review (ongoing development of IRAP tools).

A final section, entitled ‘building the climate that makes it work’, covered issues particularly important to the IRAP programme. A key concern for the regional authorities involved in the programme was that there would be widespread acceptance of the IRAP toolkit given its proposed use in the development of policy that would affect the farming practices of landowners.

The evaluation process was undertaken with an end-user group of project participants. It was based around a 2-h facilitated workshop session. Some of the group were interested in the topic, and others were vocal that it was a distraction from programme activities. Nonetheless, some concerns were identified at the session, and these were taken by the participants back to the programme governance group so that the programme could address them. Participants were particularly interested in issues regarding the IRAP programme's capacity to communicate with wider audiences. However, overall the participatory evaluation process using the checklist based on the ISKM framework was not as generative of discussion as the facilitators had hoped. In the final section, we discuss possible reasons for this and the importance of matching different types of participatory evaluation processes to the needs of programmes and participants.

Orders of Outcomes—using logic models

The Orders of Outcomes framework was developed by Olsen (2003) for coastal management, and was identified by the ICM team as being applicable to catchments. The first case-study application in New Zealand was undertaken in Auckland to evaluate ICM plans (ICMPs) in Auckland, which are used to manage the adverse effect of storm water quality, flooding and associated issues (Feeney et al. 2008). Storm water has the single biggest impact on Auckland's marine ecosystems and urban streams, which in turn has adverse impacts on the social, cultural and economic values of the regional community (Boston Consulting Group 2004). In 2007, after several ICMPs have been prepared by territorial authorities in the region, the Auckland Regional Council considered it timely to evaluate progress and sought to develop an evaluation programme using the Orders of Outcomes framework (Hellberg et al. 2009).

A logic model was developed by the ICMP work stream team that described the vision, inputs, outputs and outcomes based on an Orders of Outcome classification. The resulting model illustrates that the Auckland Regional Council ICMP team has a lot of influence over its activities to deliver the 1st Order (enabling) outcomes (Hellberg et al. 2009). This can be seen as representing programme efficiency. However, the tea m has far less control over the 2nd Order (changes in practice) and 3rd Order (harvest) outcomes. The logic model also revealed the necessary assumptions that underlie a programme seeking environmental and behavioural changes, such as the roles good relationships and shared understanding built up through personal contact play in developing high quality plans and outcomes.

The ICMP Orders of Outcome evaluation also illustrates how using a participatory process can contribute to supporting the wide engagement that is needed for successful catchment management planning. The ICMP work stream team took responsibility for populating the base model, with the external evaluation team providing the frameworks, support and facilitation as required. A number of wider meetings were held with members of other council work streams to gain their input, look for synergies and provide a communication mechanism. As Hellberg et al. (2009) acknowledge, the framework of the logic model helped to clarify and simplify the various ICMP activities. Even just the process of involving a number of staff in developing the ICMP models and asking and answering the relevant questions was useful for the council and informed the actual conducting of the evaluation (Hellberg et al. 2009). Overall, the development of the logic model offered considerable benefit in simplifying a complex programme, supporting participation, and enabling more insight and rigour to programme delivery as well as evaluation.

Social Spaces—as an evaluation & planning tool

Following the development of the Social Spaces framework, several points of reflection emerged that could be discussed by programme participants and leaders—helping increase understanding of the way to manage an ICM initiative (Table 3).

Table 3  Points of reflection emerging from the Social Spaces framework.

Space 1
Research collaboration space	How does the programme identify and promote opportunities for integrated research?
	How well recognised are the contributions of all the collaborating partners?
Space 2
Transdisciplinary space	To what extent do activities in this space promote learning, rather than just information exchange?
Space 3
Information-exchange space	What are the links between raised awareness of the ICM programme and understanding of ICM as practice?
Space 4
Intersection with the wider ICM community	Is the programme privileging engagement with the geographic community of the Motueka at the expense of the wider global and national community of interest?
ICM, integrated catchment management.

Subsequently, a participatory evaluation workshop was held to disseminate the information gathered in the initial review about communication and integration across the programme, to a wider audience of programme participants and to explore some of the issues it raised. This workshop took place during the 2007 ICM Annual General Meeting. Participants included researchers, local agency staff and members of other local groups associated with the programme. Participants first discussed the framework and used it to interpret the communication and engagement events they had individually and collectively taken part in. They next broke into groups to work up examples (tell stories) of projects or activities that they thought matched the goals and needs of each of the social spaces. Each story was summarised on post-it notes and used to populate a page for each space.

The workshop generated noticeable enthusiasm. Using the Social Spaces framework in this way provided a chance for ICM Motueka research participants to recognise the value of the work they had done, and to see the linkages and purpose behind events. It also became a way to see across the whole programme, and to make visible the intangible social connections within the programme. Furthermore, the nature of the workshop was such that it contributed to the strengthening of relationships between the ICM programme participants. By founding the workshop on ‘storytelling’ it tapped into the creative contributions of all participants, unconstrained by more formal means of information exchange.

The Social Spaces framework was a way of visualising the different types of communication and collaboration taking place in the programme—sometimes consecutively, sometimes sequentially. This visualisation helps programme participants acknowledge value in many different forms of engagement. Rather than having to choose between one approach to communication or another, each can be recognised as serving a particular purpose, meeting the needs of the important relationships in that particular social space. At the same time, it becomes possible to see any gaps that need to be filled.

Overall, the Social Spaces framework provides a useful translation of the comparatively ephemeral idea of social norms and practices into a concrete concept. Clarifying the purpose of each social space enabled people in ICM Motueka research to focus on activities that could contribute to its aims. Through the participatory evaluation, participants also increased their knowledge about social systems and interactions. However, while the idea of the existence of different social spaces is generic across any complex ICM initiative, the spaces themselves need to be checked against the reality of the individual project. For instance, initiatives without a strong research element might find the central space (Space 1) is less relevant to their experience. A participatory process with programme participants needs to begin by grounding, validating, and if necessary moderating the social spaces map.

Observations on the different participatory evaluation approaches

The three framework evaluations each used different participatory processes to support reflection. While each process had the potential to stimulate discussion among participants about different aspects of the ICM initiative, the processes used in conjunction with the Social Spaces framework and the Orders of Outcome logic model were arguably more successful than the ISKM framework evaluation in the IRAP programme. Likely contributors to this were differences in the kind of participatory inquiry, participants’ previous experience with participatory processes and group reflection, and trust and familiarity among participants. Storytelling (used in the Social Spaces evaluation) encourages appreciation and ownership, which are key factors that support good critical reflection. The checklist approach (used in the ISKM evaluation) in contrast could be regarded as judgemental within a group that was not experienced in critical reflection processes or comfortable with one another. In the IRAP exercise, the group that took part (the End User Advisory Group) had changed membership several times and were therefore less familiar with the programme and with each other, making a participatory evaluation based on critical review uncomfortable and hence less appropriate to their particular needs.

Furthermore, in ICM Motueka research, the social process specialists who worked with the framework and designed the participatory evaluation had long standing in the programme, and participants had numerous previous experiences of processes that encouraged open discussion and reflection. In the Orders of Outcome evaluation, the social process specialists were deliberately brought in to provide help and were given a respected and acknowledged mandate. The key stakeholders were also happy to be active participants in their own evaluation. In IRAP, the role of the social process specialist was not well established and hence another source of unfamiliarity for participants taking part in the ISKM-framework-based evaluation. What this suggests is that choosing participatory processes that match the skills, capacity and needs of the project at the time has an important influence on how frameworks can be used to support management of the complex social systems of ICM initiatives.

Conclusions

The d iverse communication, collaboration and engagement activities in an ICM initiative can appear messy. Managing a way through the complexity requires a balance between intuitive response to opportunity, active assessment of the social system, and monitoring and evaluation of actions. Just as biophysical models enable visualisation, interpretation and testing of hypotheses about biophysical systems, frameworks for understanding complex social interactions in ICM can be useful tools to help programme participants analyse the complex social context at the start of a project or to review what is going on as it progresses.

Any framework can provide a basis for reflection and questioning. They can be useful to ICM project leaders and participants in three ways: (1) to make sense of the social context of a project; (2) to aid design of strategies to meet social process needs such as communication and engagement; and (3) as the basis for evaluating the effectiveness of the project with a view to improving it.

Tying a framework to a participatory, developmental evaluation approach helps structure discussions about project direction to involve other programme participants. Furthermore, the workshop processes used can increase participants’ knowledge about the project, strengthen relationships and build capacity for dialogue and reflection, which are critical components of integrated environmental management. However, these evaluation and reflection processes must be designed to match the needs and capacity for dialogue of the ICM initiative at that time.

Specifically, the experience of the using the Social Spaces framework in the ICM Motueka research programme indicates its value as a tool for visualising important social connections and for understanding different communication and collaboration requirements in ICM initiatives. It was used to (1) map current engagement activities and assess their fit for purpose; (2) support participants’ understanding and appreciation of engagement activities in the programme; and (3) raise challenges to promote improved communication and collaboration.

Overall, management of the social complexity of an ICM initiative is reliant on a capacity for interpreting and managing social processes. While tools and frameworks are useful to project managers, they must be coupled with access to skills in assessment, facilitation, conflict management and participatory evaluation. This expertise is as important to ICM initiatives as more conventionally recognised capacity in biophysical science, and terrestrial and aquatic management.