Using a Delphi study to identify effectiveness criteria for environmental assessment

Abstract

Effectiveness is a long-standing issue in impact assessment (IA) practice and research – the theme is fundamental to the continued development and improvement of IA, and is essential to understanding its impacts on and contributions to environmental management. Understanding effectiveness not only requires attention to the variables that shape the context within which an IA system operates, but it also requires gauges for evaluation. In this paper, we present the results of a Delphi study conducted to develop criteria that can be used in appraisal or audits of IA effectiveness. Involving 55 IA experts, the Delphi resulted in 49 criteria, organized into nine themes. Although the framework of criteria was developed for analysis of the Canadian assessment setting, the criteria can have value to other jurisdictions. The Delphi wove sustainability throughout the criteria rather than set it as a single criterion. While context remains a key theme in evaluation, the Delphi demonstrates that there may be universal qualities that IA should possess if it is to be an effective environmental management tool, and these may transcend context.

Keywords::

• effectiveness
• Delphi method
• criteria for effectiveness
• evaluation
• context

Introduction

As a concept, impact assessment (IA) is quite simple – it is a pragmatic tool intended to identify, assess and find ways to mitigate or eliminate the potential negative impacts of proposed projects, policies or plans on the human and biophysical environment. IA has a long history. It is well established, widely practiced and is now arguably a key instrument for environmental management (Morgan 2012). Indeed, if longevity and extensive application count, IA may be one of the most successful environmental policy developments of the twentieth century, being utilized in more than 100 countries in its first three decades (Sadler 1996); as of 2013, some form of IA is mandated in 191 countries (Pope et al. 2013). Originating in the United States, by way of the 1970 National Environmental Policy Act, the expansion of the practice of IA globally resulted in increasingly different perspectives on what effective environmental IA constitutes (Sadler 1996). The diversity of opinion on effective IA is not only limited to theory, practice or politics, but it also reflects the diverse nature of geographies that pose unique challenges for IA around the world and the emergence of specializations within the practice (Pope et al. 2013). The International Association for Impact Assessment has also worked to advance the field through the development and dissemination of best practices, including methodologies and procedures that may be broadly applied to improve practice and effectiveness (IAIA 2014).

There is no lack of examples where IA has helped shape important development decisions. Then again, there are also examples where it has failed to alleviate egregious environmental impacts, although some would argue that it is more often a result of a decision-maker failing to heed the advice and information provided by an IA process. Nevertheless, a key accomplishment of IA is that it has helped ensure that, at the very least, environmental impacts are considered in the development process; however, the impact of that consideration has certainly been variable. It is not surprising then that effectiveness is a long-standing issue in IA research and practice (Chanchitpricha & Bond 2013) – the theme is fundamental to the continued development and improvement of IA and essential to understanding its impacts on and contributions to environmental management. Despite the recognition of the importance of effectiveness, the emergence of a well-developed discourse on the subject within the research and practice communities, and a growing number of systematic evaluations of the actual impacts and influence of IA on environmental quality, management and policy, criteria and conceptualization of effectiveness remain important and sometimes contested realms within IA research and practice.

In addition to the challenge of defining effectiveness, there is the ongoing tension, and often confusion, between effectiveness and efficiency. Agencies and scholars have examined the influence of individual IA components, such as public participation (O'Faircheallaigh 2010; Glucker et al. 2013), monitoring (e.g. Morrison-Saunders & Arts 2004) or social learning (e.g. Sinclair et al. 2008). Some have looked at how ‘happy’ stakeholders, notably agencies and development proponents, are with IA (e.g. Harrison 2006; McCrank 2008); these studies, often concerned about efficiency, have tended to focus on key efficiency goals such as streamlining assessment systems, making them easier for those whose undertakings are subject to IA and providing greater certainty with respect to project approval (Morgan 2012; Bond et al. 2014). However, other researchers (e.g. Morrison-Saunders 1996; Morrison-Saunders & Bailey 1999; Morrison-Saunders & Arts 2004) have specifically considered actual outcomes of IA, its effectiveness as an environmental management and protection tool, and the relationship between assessment and broad environmental management objectives.

In our research, we have sought to better understand the impact of Canadian IA processes (both federal and provincial systems) on environmental management and protection, and provide research that examines the factors that help shape its effectiveness in this regard. While our work continues, one aim is to develop a set of criteria that can be used in the evaluation and audit of IA effectiveness in different settings. To identify criteria, we consulted with a diverse group of experts drawn from public and private sector practice and academic research in Canada and in the European Union (EU) using a Delphi study design. This paper presents these results. In the sections that follow, we first present a brief review of effectiveness in IA within the context of our own research. This is followed by a description of our Delphi approach to defining effectiveness criteria and the criteria resulting from the Delphi exercise. A discussion of the results comes after, and we conclude with implications and further research needs in relation to IA effectiveness.

Effectiveness

The Canadian IA context has undergone significant change in recent years, especially at the federal level. The changes made to Canada's federal environmental assessment process have been outright decried by some (e.g. Doelle 2012; Gibson 2012) and more optimistically considered by others (e.g. Barnes & Hegmann 2013; Horvath & Barnes 2013). Improving the effectiveness of assessment was a key theme in Canada's recent federal assessment reforms, although the reforms appear to be more tailored to efficiency goals – reduce review time, narrow the range of projects and activities assessed, and create more certainty with respect to decision outcomes for project proponents. Efficiency and effectiveness do seem to be used interchangeably in the context of recent Canadian IA reform and are often coupled with the mantra of achieving greater speed and certainty. Efficiency and effectiveness in IA are not the same thing, although we do suggest that they can coexist.

The basic assumption, or perhaps the hope, about IA is that it will result in more environmentally sound development and planning decisions and better environmental management and protection than would be the case if it did not exist. But, we cannot really be sure about whether it actually achieves such goals. In a 2005 newsletter of the International Association for IA, the then president Richard Fuggle described a ‘disillusionment’ about IA ‘… and skepticism that impact assessments are contributing to better decisions’. At issue in this sentiment, which is by no means unique, is whether IA is achieving its goal. In other words, is IA really an effective tool for environmental management and protection? The relationship between IA and environmental management has indeed been examined in the scholarly literature (e.g. Bailey & Saunders 1988; Bailey 1997), but given the emerging and enduring concerns about the effectiveness of IA, the costs and time associated with assessment, and pressure to expedite development, Boyden (2007, p. 3) challenged the international IA community ‘… to better identify the benefits as well as costs of impact assessment’. It is fair to say that this concern has hardly diminished; it is acknowledged more broadly and there has been considerable thought given to how effectiveness can or should be defined, the importance of context, and the evaluation and measurement of the effectiveness of IA processes and outcomes.

A fundamental ideal of IA is that it should be preventative and, as such, is well positioned to be a more effective instrument for protecting the environment from the impacts of development, or at least mitigating them, than many concurrent or reactive environmental management tools (Hollick 1981; Gibson 1993; Wood 1993; Senécal et al. 1999; Snell & Cowell 2006). As part of advancing this ideal, general guidelines have been put forward for the development of IA practice (Snell & Cowell 2006; Duinker & Greig 2007; Fitzpatrick & Sinclair 2009; Retief et al. 2011; Canadian Environmental Assessment Agency 2013), its theory and purpose have been examined (Cashmore 2004; Chanchitpricha & Bond 2013), and strategies for auditing IA performance advanced (Wilson 1998). IA has also been assessed with regard to the types of regulations that affect it (Sandham et al. 2013), as well as the commonly cited critical factors for implementation (Zhang et al. 2013). Identifying the potential consequences of current or proposed actions can also be found in international discussions about appropriate indicators to measure, analyze and assess the potential impact of IA. Discussions of this nature are becoming increasingly common as concepts of sustainability and development impacts are being evaluated from broader, ecosystem-based perspectives (e.g. Moldan et al. 2012; Morrison-Saunders & Retief 2012; White & Noble 2013). As assessments often require public participation, discussions about facilitating meaningful participation and critical education have also gained traction as a means to broach IA effectiveness (Sinclair et al. 2008; O'Faircheallaigh 2010; Glucker et al. 2013). Baker and McLelland (2003), for example, evaluated effectiveness from the perspective of aboriginal involvement, seeing this as a key characteristic of effective and successful IA within a complex land-use and land-claims setting. Other bodies on knowledge, outside the IA field, also offer theories and methods relevant to the evaluation of IA. For instance, Van Doren et al. (2013) draw on planning theory to consider the substantive effectiveness of strategic assessment and develop an approach to evaluation. These conversations about effectiveness often relate to the validity and refinement of the approach to, and practice of, IA.

Sadler (1996, p. ii) suggests that the real test of effectiveness may be whether IA has demonstrably affected decision-making and supported environmental management objectives within the institutional framework that it operates. Rather than setting out specific requirements, Sadler proposed appropriate timing, clear and specific direction, quality information and practice, and receptivity of decision-makers as the four key components that result in effective IA application (Sadler 1996). Bond et al. (2013) proposed an alternative framework for IA effectiveness assessment, which reinforces the value of Sadler's (1996) procedural, substantive and transactive qualities, and also sees normative objectives, knowledge and learning, and pluralism as key components. The latter approach builds on the former, while facilitating an understanding, and incorporation, of the diverse opinions or approaches involved in IA, and builds in a component for reflexivity.

Recent reviews seem to suggest that effectiveness of IA is best analyzed at the state level, evaluating the validity and impact within the country-legal framework that it operates (e.g. Appiah-Opoku 2001; Pölönen 2006; Heinma & Poder 2010; Toro et al. 2010; Che et al. 2011; Pölönen et al. 2011). However, specificity of this nature, resulting in the widespread use of a method without broadly applicable measures of effectiveness or necessarily evaluating the success of IA not only as a concept and practice realm, but as an environmental management tool, may be one of the reasons effectiveness discussions continue to be prominent in the field. The introduction of IA as a process that influences decision-making can significantly impact proposed and approved activities. This influence may take the form of the rejection of a proposed activity, the identification of steps for mitigation and the discouragement of the submission of environmentally unsound proposals (Pope et al. 2013). Some suggest, however, that in practice and despite expectations, IA has not been able to minimize impacts, avoid irreversible impacts and facilitate sustainable development (Cashmore et al. 2004). Other criticisms point out that the benefits of IA are declining due to pressures, or reforms, such as streamlining review processes (Morgan 2012; Bond et al. 2014).

In light of less than desirable outcomes in some contexts, suggestions for assessment reform have been proposed which adopt a more determinative role for IA in decision-making (Jay et al. 2007). The impacts of context, especially professional and organizational cultures (Emmelin 1998a, 1998b), and the significance of politics and politicization of IAs are increasingly recognized, and some efforts are beginning to outline processes whereby these contextual issues can be integrated in order to advance effectiveness evaluations (Cashmore et al. 2010). It is not that effectiveness studies of IA have not been done. Rather, such studies have often focused on procedural effectiveness; that is, the nature and efficiency of IA frameworks (Harrison 2006) and the extent to which IA applications integrate such aspects as cumulative effects (Ball et al. 2013), or human health (Bhatia & Wernham 2008), how the significance of environmental effects is determined (Lawrence 2007), or the actual quality of the environmental impact statement (Tzoumis 2007). In those instances where attention has been directed toward functional aspects of IA, the focus has been on particular IA dynamics, such as social learning (Diduck et al. 2013), consideration of alternatives (Steinemann 2001), or follow-up and monitoring (Noble & Storey 2005). Effectiveness research needs to bring together these themes, but it needs to do so in a manner that is pragmatic and useable for evaluation and policy audiences, and continues to support what research has noted are the value contributions that IA makes to environmental management and protection (e.g. Bailey & Saunders 1988).

Using a Delphi approach to define effectiveness criteria

The Delphi method is a structured, iterative consultation and survey process that typically includes two or more stages or rounds. It employs a systematic and interactive approach to asking a small group of experts in a specific field to forecast an outcome and/or estimate unknowns, for the purpose of generating opinions to help reach decisions (Hung et al. 2008). Although commonly used to produce or create consensus (e.g. Graham et al. 2003), the original aim was not to do so (Linstone & Turoff 2011).

The Delphi has been used in research and practice as a tool to effectively engage diverse stakeholders (Geist 2010). Although originally developed and utilized by the RAND Corporation in the 1950s for the purposes of forecasting and prediction, its application has expanded greatly and it is used in many disciplines (Gupta & Clarke 1996). The Delphi method has been used to study varied and complex questions, ranging from understanding risk (Webler et al. 1991), financial markets (Kauko & Palmroos 2014) and social support policies (Yap et al. 2014).

The structured approach of the Delphi is generally considered a more accurate approach to forecasting and consensus building than unstructured alternatives (Rowe & Wright 1999), although this has been contested (Woudenberg 1991). Among criticisms of the Delphi method are that the resulting expert consensus is the product of conformity and group pressure (Woudenberg 1991), or potential desirability (Ecken et al. 2011), and that participant selection and attrition over the Delphi rounds may result in a nonrepresentative consensus (Bardecki 1984). Anonymity and controlled feedback were advocated by early adopters of the method as ways to reduce the potential impacts of group pressure, and careful consideration by the researchers of opinions can ensure that opinions different from the majority are indeed accounted for in the analysis and results (Pill 1971). Although techniques such as shared written rationales can inform others of the reasons for taking a certain position, majority opinion may influence the minority (Bolger et al. 2011). Thus, defining whom to include as the expert in a Delphi is thus an important consideration (Okoli & Pawlowski 2004). The Delphi method can be an effective research approach and adapted for diverse contexts (Landeta 2006; Banuls & Turoff 2011; Hasson & Keeney 2011; von der Gracht 2012), such as IA where the field of practice is broad and draws from multiple disciplines and sectors.

Our Delphi application sought the advice of a group of IA experts to help develop criteria for evaluating the effectiveness of an IA system. The Delphi consisted of three stages and was based on one concise question: What criteria should be used to evaluate the effectiveness of an IA system? In the Delphi, we defined the effectiveness of IA as the extent to which it identifies, assesses, and finds ways to mitigate or eliminate the potential negative impacts of development, and importantly how well it helps protect or improve environmental management and ultimately the state of the environment. The question was open-ended and required text or qualitative responses whereby participants were asked to propose, and define, effectiveness criteria. We used the term ‘impact assessment’ to capture the range of assessment processes, including environmental, strategic and social, and adopted a broad definition of environment including both the biophysical and human realms (social, cultural and economic).

The expert group was identified by selecting key researchers in the field, key government and public agency personnel, and a list of the EU national experts at the time. We chose to include a group from the EU because the diversity of systems and multiple interactions has relevance to the Canadian experience and because connections between IA research and practice are well developed in the EU. Although situated in different jurisdictional contexts, the make-up of the group was also helpful in overcoming potential language and culture challenges that researchers have noted can affect the performance of a Delphi study (e.g. Hung et al. 2008).

The Delphi group embodied a key characteristic of IA practice – it is an interdisciplinary or multidisciplinary field that draws knowledge broadly from the sciences and social sciences, and practitioners span the public and private sectors and academe. We invited 214 people to participate (100 from Canada and 114 from the EU). Fifty-five accepted and participated in the first round of the Delphi, and at the end of the third round, there were 44 Delphi members. The attrition level is considered modest. Member's names were not shared within the Delphi group and we assured anonymity when presenting the results. Participants were split relatively equally among government (national, regional and provincial), academic, and private sector and international or transnational organizations (e.g. the European Commission and the United Nations). About 38% were from government and international organizations, 28% academic and 23% private sector. There were no conservation or environmental organization participants. This reflects the nature of the original invitation list, which was mostly government, academe and private sector practitioners, and a reality that we encountered where nongovernmental organizations tended only to interact with an assessment process on a project or issue basis; there were few such organizations with an ongoing and broad interest in the assessment process. The Delphi is also a method that is to a great extent dependent on those willing to participate and devote time to the process. However, some respondents reported that they also worked for such organizations, but not on a full-time basis. Some academic and private sector respondents also noted that their work crossed sector boundaries (e.g. a consultant who also works as a sessional lecturer or an academic doing contract work for a consulting firm).

A pretest of both the stage 1 Delphi question and the working definition of effectiveness was also conducted. The pretest readers suggested that IA be approached as a linked system and thus the Delphi should be open to comments relevant to system-level qualities (process/procedural aspects) and their links to decisions (substantive outcomes). The pretest readers also suggested that comments on project or plan level of application be encouraged, and that environmental management be conceptualized broadly and include protection and improvement. We did not explicitly include process efficiency, but aspects of this quality, or objective, did emerge throughout the Delphi and are evident in some of the resulting criteria.

Responses were collected using an online survey design tool accessible only to the participants. The criteria proposed by the group were qualitatively analyzed by four researchers at the end of each stage to identify both unique and common themes. The resulting list was then circulated back to the respondents for refinement and change. Further analysis and modification followed until the final list was developed. Minor editing and refinement of wording was made to the final list based on a review of comments from the Delphi group across the stages. The process was a modified Delphi – it sought relative consensus, which means that some of the criteria reflect a majority agreement. The process took 18 months to complete.

Criteria for effective IA

At the end of stage 1, there were 25 broad criteria identified by our Delphi participants; by the end of stage 2 this had been refined to 16 themes; and after stage 3 we were able to derive a final list of nine criteria themes. Some criteria were specific, and some had multiple components. In stages 2 and 3, there were instances of individual change, where a participant would propose a criterion in an earlier stage, and then ask that it be removed later, or reject the refinement or rewording. This supports the value of the Delphi as a place where ideas can be not only identified and refined, but changed or even rejected, by those who initially proposed them, as participants are exposed to other ideas, participate in argument or see the notion juxtaposed with the contributions of others. It is a reflexive process capable of also serving both knowledge building for a community of practice and collaborative learning functions.

The final criteria are grouped under nine criteria themes – a framework for evaluation of effective IA. These are outlined below. In the last stage, we asked participants to identify the four themes they thought were the most important. We did not ask them to rank these. The four most commonly identified effectiveness criteria, by number of votes, were stakeholder confidence, integrative and linked to decision-making, promotes long-term substantive gains in environmental quality and comprehensiveness. Some participants noted in their comments that while they valued ‘strong’ participatory qualities, they believed that these should also be considered a natural asset for building confidence in the IA process or system, or that participatory qualities were an aspect of one or more of the other criteria they chose.

1. Stakeholder confidence

• The IA process is known by stakeholders to be objective, and there is confidence that other processes do not predetermine the IA decision.

• The process is understood by stakeholders, and information about the process, proceedings and its authority is accessible and clear.

• The intent of the process is acknowledged and clearly stated, whether it is to advise, decide, or to only identify baseline conditions and determine impacts.

• There is confidence that major projects or powerful proponents cannot circumvent the process (see also 4(b),(c)).

2. Integrative and linked to approval decision-making

• The results of the IA process are clearly accounted for in the decision (the eventual approval, rejection or approval with conditions).

• The process demonstrably informs, and the results are integrated into, other subsequent or coincident environmental approval and review processes.

• There is capacity to integrate the knowledge and results of other processes into the IA process without unduly influencing its outcomes (see also 1(a)).

• An initiative may not proceed through other approval processes or receive other approvals until the IA process is complete and the initiative approved.

• The process considers impacts beyond the immediate time scale of the policy, plan or program, when applying strategic assessment.

3. Promotes betterment and longer-term and substantive gains to environmental management and protection

• The IA process and its outcomes minimize or eliminate adverse environmental effects that may result from the initiative.

• The process seeks betterment of the environment, when possible, by ensuring net benefits to the environment.

• The process seeks to identify and sustain social and biophysical systems that interact and may be affected by assessment-subject activities.

• The process prevents imposition of significant adverse effects onto future generations.

• There is mandatory follow-up and monitoring, including a supporting audit and public reporting system to ensure compliance with approval conditions.

• The process provides follow-up provisions to assess the efficacy of mitigation requirements and reports on environmental benefits (e.g. provision of compliance schedules, mitigation reports and post implementation audits, evaluation of immediate and longer-term gains to environmental management and protection).

4. Comprehensiveness

• The definition of ‘environment’ and ‘environmental effects’ encompasses social/cultural and ecological/biophysical factors and their interrelationships at multiple scales.

• IA is applied to the range of initiatives/activities that significantly affect the environment, whether the proponent is from the public or private sector (see also 1(d)).

• Initiatives may be ‘screened out’ (exempted from IA) if there is sufficient information to determine that impacts are insignificant, or otherwise addressed by an alternative process, but listed exemptions are limited to emergency or similar initiatives (e.g. urgent flood control works).

• There is a mandatory scoping stage that occurs early in the assessment process to focus the assessment on key issues and identify opportunities for environmental protection and improvement, and there is opportunity to deal with new information or issues identified throughout the assessment process or during project implementation (see also 3(e),(f) and 9(e)).

• The process requires identification and reasonable consideration of alternatives, including ‘alternatives to’ the initiative and ‘alternative means’ of carrying out or implementing the initiative.

• The process assesses cumulative effects.

5. Evidence-based

• The decisions that follow the IA process clearly and directly reflect the evidence presented in the assessment and/or review proceedings, and the process is open to hearing and considering all relevant and opposing evidence.

• Uncertainties and assumptions about data, system behaviors and future conditions are disclosed and acknowledged in the decision.

• Impact predictions are formulated in such a way that they can be tested or used for follow-up.

• The process requires monitoring and follow-up, and the data and reporting from those activities are made publically accessible and retained for use in subsequent assessments and decision-making processes.

6. Accountability

• There is a requirement for regular, independent public review of the assessment system, its performance and effectiveness (e.g. a five-year review of process, legislation and regulations).

• Documentation and information disclosure requirements are binding on the process and its administrators, proponents and all other stakeholders.

• There is open and easy access to timely, accurate and full and complete information early and throughout the assessment process through formats that provide extensive access and acknowledge different forms of access need (multiple formats are used: electronic, print, languages, verbal and other).

• The process is independent, and where needed multidisciplinary organizations exist to hear requests for exemptions and inclusions, conduct hearings (when they are required) and review assessment documents and reports.

• Roles and responsibilities in the assessment, review and decision-making processes are clearly identified.

• Roles and responsibilities for post-IA, including implementation of the initiative and follow-up on mitigation and monitoring and reporting, are clearly identified.

7. Participation

• There is a requirement for stakeholder participation throughout the process.

• Participation opportunities are made well known, while recognizing that such engagement will vary in scale and method according to the nature and scale of the initiative being assessed, the stage of the process and the social–cultural context.

• Sufficient resources and time are provided to support participation.

• The participation approach is designed to improve the quality of the proposal, affect the assessment and influence the decision.

• There is a requirement to broadly consider, use and respect multiple forms of knowledge where applicable and available (e.g. scientific, applied-technical, aboriginal, local and culture-specific).

• Hearings and other similar deliberations are open to the public and there are no unjustified limitations to open deliberation and presentation of evidence (whether through the imposition of place, time of day, time allowed, insufficient resources, or cultural or social barriers, or other unwarranted limitations).

• There is a requirement to publically report on stakeholder engagement, including how it was undertaken and what was said, and how it was accounted for in the assessment and decision.

• There is a requirement to explain how participation was accounted for in the decision.

8. A legal foundation for IA

• IA must be codified in law.

• The legal foundation for IA must provide clarity for stakeholders with respect to applicability, assessment requirements, disclosure requirements, and process components, reporting and decision-making.

• The process contains a legal base for participation and accountability requirements.

• The process must outline provisions for enforcement and addressing with noncompliance with assessment requirements or subsequent decisions.

• The IA system must provide decisions (for approvals, conditions, rejections, exemptions and inclusions) that may be appealed by stakeholders or other affected parties based on questions of process veracity or interpretation of law.

9. Capacity and innovation

• The IA process must be administered by competent and impartial authorities with sufficient staffing, skills and qualifications to administer the process, and to review and evaluate technical, social and scientific data.

• The process must provide sufficient financial resources to review agencies to ensure the integrity, effectiveness of, and confidence in, the process.

• Mechanisms exist in the process for the early consideration of assessment-subject initiatives and the provision of advice to proponents.

• Information accessibility and participation are enhanced by the use of innovative technologies and formats for communication, stakeholder capacity building and information access.

• The process and the supporting institutional framework are flexible, adaptive, and open to new and innovative tools and approaches to assessment and evaluation.

Discussion

After the criteria were derived, we were interested in knowing the extent to which the criteria resulting from our process would differ from or reflect effectiveness themes already present in the IA literature. With some notable exceptions (e.g. Sadler 1996), effectiveness criteria have commonly been based on opinion, albeit well-informed, reflective and experiential ones. Using the Delphi to generate such criteria, however, produces a relative consensus that is the outcome of the practice and research experience of group members, tiered and interactive consultation, and deliberative argument.

Some of the criteria that emerged from our study do overlap with parts of existing scholarly lists (e.g. Gibson 1993; Sadler 1996; Senécal et al. 1999; Wood 2003); indeed, some participants noted that they were reflecting on ideas from such sources, or from their own writings. The reiteration of some criteria reflects the strength and established nature of key ideas and tenets of assessment practice. But in our results, several differences do emerge. The ranking of criteria themes indicated a preference for four IA qualities: stakeholder confidence, integrative and linked to decision-making, promotes long-term substantive gains in environmental management and protection, and comprehensiveness – this provides some indication of priorities in conceptualizing effectiveness, at least for this Delphi group.

The Delphi group identified participation as a criterion for effective IA, and it is a prevalent theme in other criteria lists (e.g. Gibson 1993; Sadler 1996), but the majority of participants chose not to include it as one of the four most important effectiveness themes. Perhaps, the expectation of a participatory process may be relatively well advanced in democratic developed countries, which helps in the evaluation of Canadian IA, but there are many nations without substantive public participation mechanisms built into their IA systems. We support the contention that an effective IA process includes public participation mechanisms, and at a basic level these should meet the criteria noted here. The design and implementation of such tools need not mirror those prevalent in western settings, but should respond to specific social–cultural settings while providing the best opportunities for hearing from those impacted by development.

The role of stakeholder confidence is perhaps a unique theme in this study, although hints of this criterion are observable in other rubrics – for example, Wood's (2003) criterion ‘where the benefits of assessment are seen to outweigh the financial and time costs’. Regardless of the extent to which the themes and criteria outlined here reflect recognized criteria, the Delphi provided a setting within which ideas were debated and modified, or even removed, by colleagues through stages of discussion. The resulting Delphi criteria presented here are also aimed at evaluation and audit application, which gives them perhaps a more detailed quality than what may be present in other, more broadly worded criteria or principles.

Context

Two specific issues emerged from our Delphi that are important to understanding and advancing effectiveness criteria for IA evaluation. The first concerns the role and importance of context. Given the importance of context in discussions of IA effectiveness, it is not surprising that context was a theme that emerged in the Delphi. In recent years, both effectiveness and context have emerged as important ideas in evaluating IA systems and outcomes. The general argument about context is that what is effective in one context or in one regulatory system, or resource sector, may not be considered so under another. Furthermore, understandings and interpretations of effectiveness may vary from the proponent to the regulator to the general public (Hilding-Rydevik & Bjarnadóttir 2007). In this study, context emerged as a quality that certainly shapes effectiveness. However, while context is important for understanding the reasons that IA may or may not improve environmental management, advance sustainability or be better connected to decision-making, it is not an excuse for a less than effective IA system – nor does it preclude the evaluation or audit of IA processes for effectiveness using generically accepted evaluation criteria.

We agree that gauging effectiveness is possible and essential, and that such evaluation must also acknowledge contextual variables such as institutional and organizational cultures, the values and doctrines of professions, and political–jurisdictional qualities. Methodologically, these are qualities that are also relevant to participants in this study. However, regardless of context, we propose that there are universal qualities that IA must possess if it is to be effective as an environmental management tool, and the criteria we provide here is a starting point for articulating and evaluating those qualities. For some Delphi participants, the process resulted in reflection on the qualities of the IA systems in their jurisdiction. One Delphi member commented that, after reviewing the criteria, they ‘could only conclude that their IA process was ineffective’, another noted that if they ‘applied the resulting criteria to their system it would fail the effectiveness test’. It was evident in the responses that there was a clear struggle with the articulation of criteria that could be broadly applied rather than a set that was contextually certain.

Sustainability

A second issue concerns how sustainability, or contributions to sustainability, can be addressed in IA effectiveness criteria. We did not ask participants to comment on sustainability, but it emerged as a theme in stage 1. It is an understatement to say that sustainability is an important term in environmental management. Nonetheless, despite the wide adoption of sustainability as a semantic ingredient in the objectives, mission statements and mandates of a range of organizations that deal with IA, it is also seen as lacking clear meaning, possessing conflicting qualities and being difficult to put into practical use (White & Noble 2013). However, Cashmore et al. (2004) propose that the contribution of IA to sustainable development, while unknown for sure, is perhaps more than may be typically assumed. The contributions are indirect, but through stakeholder involvement and the subsidiary impacts of IA on institutional actors (government, industry and science), the process can shape development outcomes by influencing causal pathways to decision-making. This impact is evidently more coincident than deliberate (Cashmore et al. 2004).

Some in the Delphi group viewed sustainability as an important word that should be acknowledged, and others saw it as a difficult notion needing substance, an understanding of institutional settings, clear pathways of influence and discernable operational characteristics. Other Delphi members emphasized the importance of defining the sustainability in a more operational way, while some wanted it dropped altogether rather than repeating ‘commonly used definitions’ and ‘prosaicisms’ associated with the concept, as two members noted in their comments. At the end of Delphi stage 1, there was a stand-alone sustainability criterion. Sustainability has been linked to IA in no small part because sustainability may be an idea in search of a practical means of implementation; IA is seen as offering that attribute. There is also an emerging sustainability assessment field that has sought to formalize the concept within IA practice and conceptual frameworks (Bond et al. 2012; Morrison-Saunders & Retief 2012). However, the functional qualities of IA cannot compensate for the ethereal nature of sustainability. Because of imprecise and sometimes contradictory interpretations of its meaning, in subsequent Delphi stages the group moved away from a single sustainability criterion and instead integrated what may be its most tangible quality, a holistic and necessarily linked perspective on environment and institutional arrangements, into several of the proposed evaluation criteria – integrative, comprehensiveness and promoting betterment.

The Delphi facilitated the adaptation of arguments to incorporate sustainability across several criteria rather than either discarding it, or seeing it as a stand-alone measure. The integrative and linked nature of sustainability and its broad qualities may be better served by incorporating the palpable objectives of sustainability into multiple criteria. Regardless of the challenge in accommodating a theme that can be equivocal, the Delphi was able to find a balance and articulate qualities that are pragmatic and relevant to practice and evaluation, indicate the greater need and potential of IA to contribute to sustainability by using and shaping causal pathways, and at the same time be reflective of the broad objectives of sustainability.

Conclusions and further research needs

This paper presented an expert-based set of criteria for analyzing the effectiveness of IA, derived based on a Delphi process. Environmental assessment should be a vital public policy instrument for ensuring good environmental management, and if applied with sincerity it can improve the quality of development decisions and help decision-makers avoid unwarranted impacts. The criteria reflect a combination of operational, process and institutional factors that shape the impact and effectiveness of IA application and practice. The inference may be that the effectiveness of procedures and processes, including the institutional environment in which those procedures and processes operate, is a substantive part of ensuring that IA is an effective tool for environmental management. The resulting framework helps advance the discussion of effectiveness and provides a foundation for the evaluation of both operational and conceptual–theoretical qualities of IA practice and implementation, and environmental management outcomes and effectiveness.

Although we did not set out to explicitly test the Delphi as a research instrument, in our experience the Delphi functioned as a valuable tool for developing criteria for evaluation and audit of processes and systems, and for reaching relative consensus about attribute, objectives and broad metrics. In this study, there were changes in opinion, and the rethinking of individual stances and positions among the stages, which can be an indicator of reflective thinking and social learning emerging as participants saw their contributions merged and modified, and had the opportunity to do the same with the ideas presented by others. But this potential quality also runs the risk of focusing on those qualities that can be broadly agreed upon while possibly omitting important but less generally accepted criteria. For practitioners, the Delphi can be an important tool in IA research and evaluation for understanding the qualities of process, identifying regulatory change needs, predicting system impacts and performance issues, and forecasting the likely outcomes and environmental management contributions of policy or process reforms.

The study also meets, in part, the intended outcome of providing criteria (grouped under nine criteria themes) for use in the evaluation, and audit, of IA processes and the criteria may be framed as evaluation or audit questions. The next phase of this research will be to develop measures or metrics for each of the proposed criteria, which can be used in the evaluation or auditing of IA effectiveness. While there is a need to develop qualitative and quantitative evaluation approaches, addressing the methodological and analytical needs of evaluation is more challenging for some of the criteria than for others. This suggests the need to also draw on the theories and methods advanced by fields outside IA, such as for policy analysis, planning or program evaluation. Regardless, there may be universal qualities that IA should possess if it is to be an effective environmental management tool, and these could well transcend context. This study demonstrates that there is the potential to develop general criteria for IA effectiveness. IA in many jurisdictions is being reconsidered in its present form as decision-makers and other stakeholders seek to evaluate the role of IA in development processes, planning and environmental management. The objectives of these deliberations may be environmentally altruistic, or not, but if anything these pressures reinforce the importance of understanding and demonstrating the effectiveness of IA as an essential part of environmental management and protection.

Acknowledgements

The authors thank the Delphi study participants for their valuable contributions, and especially their time and patience. The advice and insight of the reviewers and IAPA editors are also greatly appreciated.

Additional information

Funding

The Social Sciences and Humanities Research Council of Canada provided funding for this research. This support is gratefully acknowledged.