Developing key performance indicators for strategic environmental assessment effectiveness: a systematic framework

Figures & data

Figure 1. Flow diagram of the methodological bottom-up (1 to 3) and top-down (4 to 7) steps for sieving international SEA effectiveness indicators to select Key Performance Indicators for Ireland. ‘n’ refers to the total number of indicators at that methodological step, with specific numbers provided for each SEA effectiveness dimension.

Table 1. SWOT summary of the findings of the second Irish SEA review (Based on EPA 2020).

Strengths Robust context: effective transposition of legislative requirements; appropriate guidance; strong role of the EPA; strong input by statutory consultants; SEA forum. SEA perceived as helping to promote sustainable development. Improving procedural practice over time. Environmental reports generally meet legal requirements. Geographic Information System (GIS) SEA-relevant information database (www.enviromap.ie). SEA benefits perceived as outweighing costs.

Weaknesses Limited (efforts for) public participation in SEA. Limited consideration of alternatives. Shortcomings in mitigation and monitoring measures. Poor reporting of SEA influence/changes in the SEA Statement. Data gaps for some environmental themes. Poor consideration of health and social issues in SEA. Difficult to tell if plan changes are due to SEA or other reasons. Some lack of resources/training.

Opportunities Future rounds of cyclical land-use plans and non-land use plans (e.g. forestry, tourism) give opportunities for improved SEA practice. Application of good practice guidance (e.g. alternatives, monitoring). SEA research and SEA Forum for knowledge exchange. Increasing public and decision-makers’ awareness of environmental problems (e.g. climate change, integrity of sites of international nature conservation importance).

Threats Personnel changes within SEA and plan-making teams, loss of institutional memory. Strategic/technical nature of some plans not conducive to SEA. Lack of public (online) availability of SEA documentation affects knowledge and learning from SEA. Non-engagement by the public even when consultation opportunities are available. Level of openness of planners and decision-makers to integrating environmental information and recommendations. Difficulty of quantifying the benefits of SEA reduce understanding of these benefits.

Table 2. Strategic, tactical and operational goals of SEA (from the literature and brainstorming by the project team).

Strategic goals: Plans that actively promote a robust approach to sustainability and provide a high level of protection to the environment. Planners and politicians who understand the value of SEA and environmental protection. SEAs that are no longer and more costly than necessary. Learning from the SEA process is captured and passed on for future SEAs and project environmental impact assessments. Tactical goals: Identify/clarify environmental and social objectives that SEA should be helping to achieve, and raise the profile of environmental constraints (sensitive habitats, greenhouse gas emissions etc.) Train planners and politicians with regards to SEA and especially on the need to start early and be open to the learning and outcomes of the process. Improve SEA practitioner capacity, especially focusing on how to make SEA more efficient. Ensure that SEAs focus on alternatives and mitigation measures. Make GIS data easily available to SEA and plan-making teams. Inform lower tier impact assessments (tiering). Operational goals: Carry out a legally compliant SEA that is not prone to legal challenge. Use the SEA to make the plan as reasonably sustainable and environmentally friendly as is possible given the context in which the SEA is being carried out. Use the SEA to explain how to support increased understanding of the local environment, avoid future environmental risks and ensure transparent decision-making. Be clear about preferred alternatives and recommended mitigation measures, including links to lower tier assessments. The planning team implements as many of the preferred alternatives and mitigation measures as possible. Include monitoring commitments and resources into the plan for future learning. Make the SEA environmental report concise and easy to read.

Table 3. Elements of good/best SEA practice identified by interviewees (n.B. At this stage of the interview, respondents were not explicitly thinking in terms of effectiveness dimensions as the question on SEA effectiveness dimensions followed from this, but the responses have been categorised here by the authors to reflect the research framework).

SEA effectiveness dimension	Responses (number of respondents in brackets)
Context	Use of SEA guidance and understanding/correct interpretation of SEA Directive’s requirements; consistency with international guidance (3). Openness of plan-makers to information sharing and SEA recommendations (5).
Procedural	Robust, up-to-date baseline information/database; technical rigour; use of specialists where needed (7 respondents). Good practice screening, scoping, alternatives, impact assessment, mitigation and monitoring (5). Viability of alternatives (2). Good understanding of the plan (1). Readable non-technical summary (not cut and paste from main report) (1).
Pluralist	Good public participation: early and interactive opportunities for different parties to bring information to the table, and written feedback on all public comments (2). Effective and timely consultation with relevant environmental authorities and the public (4).
Substantive	SEA contributes to environmental integration into the plan, informs decision-making (9). Adequate implementation of the plan; monitoring and adaptive management (3).
Normative	Plan contributes to sustainable development, achieves environmental outcomes (2). Plan is beneficial for the community (1).
Knowledge and learning	Effective monitoring which actively influences the next cycle of plan-making (4). SEA helps to provide transparent information on impacts, alternatives and mitigation (2). SEA helps to build capacity of the stakeholders (2).
Transactive	None

Figure 2. SEA effectiveness dimensions identified by interviewees and survey respondents.

Table 4. Most relevant indicators for measuring SEA effectiveness according to international survey respondents (n.b. The related effectiveness dimensions were not included/presented in the online survey).

SEA effectiveness dimension	Elements of SEA effectiveness*	% Responses
Context (3%)	Level of information sharing amongst actors	3%
Procedural (28%)	Identification and description of reasonable alternatives and justification of the selected alternative against environmental objectives	10%
	Monitoring programme includes clear frequency, responsibilities, thresholds/triggers for remedial action and reporting periods	9%
	Identification of cumulative effects	5%
	Date when the SEA starts vis-à-vis the plan start date	3%
	Number of meetings between the SEA team and the plan-making team	1%
Pluralist (20%)	Level of public input into alternatives and mitigation measures	7%
	Extent of ‘two-way’ communication of the public participation approaches/techniques/methods used	6%
	Integration of public input into SEA documentation	4%
	Number and representativeness of participants	3%
Substantive (22%)	SEA impact on the plan/programme (i.e. integration of SEA recommendations into the plan)	8%
	Appropriateness/comprehensiveness of mitigation measures and their incorporation into the plan	7%
	SEA recommendations ownership/openness to integration by plan-makers and decision-makers	7%
Normative (5%)	Level of contribution towards sustainability in light of major environmental problems/challenges	5%
Knowledge and Learning (19%)	Extent of monitoring, and monitoring data made publicly available	9%
	Public availability and quality of SEA documentation	5%
	Extent of learning from the SEA process (planners’ perception of their understanding of the SEA process at the beginning and the end of the process)	4%
	Availability and take-up of SEA training/courses by SEA actors	1%
Transactive (2%)	SEA preparation and implementation costs	1%
	SEA preparation and implementation benefits	1%

*Not all of these fit neatly into a given SEA dimension. For instance monitoring being made publicly available could be considered pluralist (public availability of data) or knowledge and learning (data to improve future rounds of SEA). Also the different number of questions under each SEA effectiveness dimension (e.g. 5 for procedural, 1 for normative) could affect the overall number of responses related to each dimension. Nevertheless, a clear trend emerges from the responses.

Table 5. Key Performance Indicators (KPIs) of SEA effectiveness in Ireland.

SEA effect. dimension	Key Performance Indicators			Data source
			SEA Rept.	SEA Statement*	Monitoring report	Plan	Planner Interviews
Context	1	SEA documents’ easy availability on a public website	√	√	√
Procedural	2	Consideration of realistic and appropriate ‘within plan’ alternatives	√
	3	Assessment of cumulative impacts of the plan plus other plans, projects and external trends	√
Pluralist	4	Environmental authority recommendations taken on board (either integrated into the plan or a reason given that directly responds to the concerns, where the concerns are within the ambit of the plan)	√	√		√
	5	Public recommendations on SEA documents taken on board (either integrated into the plan or a reason given that directly responds to the concerns, where the concerns are within the ambit of the plan)	√	√		√
Substantive	6	Changes made to the plan in response to proposed SEA mitigation measures	√	√		√
Normative	7	SEA contribution to environmental improvement (focus on key impacts of the plan, tests against relevant environmental targets, leads to plan changes towards achieving environmental targets)	√	√			√
Knowledge and Learning	8	For cyclical plans, SEA monitoring carried out for the previous cycle of the plan, and monitoring findings referred to in the current SEA			√
	9	Planning team documentation of lessons learned from this SEA and suggestions for improving the next round of SEA	√	√			√
Transactive	10	Planning team documentation of the costs and benefits of SEA, and what can be done to improve its benefits		√			√

*Article 9.1b of the European SEA Directive requires the preparation, after plan adoption, of an ‘SEA Statement’ that describes the influence of the SEA process on the plan.

EPA. 2020. Second review of strategic environmental assessment effectiveness in Ireland. Report Nº 306. EPA Research Programme 2014–2020. [accessed 14 Sep 2023]. http://www.epa.ie/publications/research/environmental-technologies/Research_Report_306.pdf.

 Google Scholar