Strategic environmental assessment in the electricity sector: an application to electricity supply planning, Saskatchewan, Canada

Abstract

A strategic environmental assessment (SEA) framework for electricity sector planning is developed and applied to evaluate electricity supply scenarios for Saskatchewan, Canada. The overall goal of the SEA application was to identify a preferred future electricity production path, demonstrate the application of a quantitative SEA process that operationalizes sustainability principles through the use of assessment criteria, and examine the methodological implications resulting from the application of a structured SEA framework. Results of the application identified a renewables-focused electricity supply preference, but with several implications for electricity sector investment and sustainability, including increased infrastructure requirements and increased cost of electricity. Results also demonstrate a practical approach to the operationalization of sustainability through the application of assessment criteria that are linked to higher level principles. The use of structure in the SEA process provided for replicability, transparency and the ability to quantify issues of uncertainty in Plan, program and policy (PPP) decision-making, while at the same time maintaining flexibility to tailor the SEA framework to the electricity sector context.

Keywords:

• strategic environmental assessment
• sustainability
• electricity sector planning
• multi-criteria analysis

Introduction

Strategic environmental assessment (SEA) methodology has advanced considerably over the past decade. Some have argued that good SEA methodology is flexible to context (Retief 2007; Nilsson & Dalkmann 2001; Partidario et al. 2008); others have cautioned that, in being flexible to context, both structure and consistency must be maintained (Gunn & Noble 2009; Therivel 2010; Browne & Ryan 2011). Flexibility is a defining principle of SEA, if not one of its strengths; however, methodological flexibility can pose significant challenges to both the practitioner and the decision-maker (see Liou et al. 2006; Noble et al. 2012).

The aim to ensure flexibility in SEA has resulted in guidance that is often too generic, leading to criticisms of SEA as ad hoc, vague or inconsistent (see Auditor General 2004; Retief 2007), treating SEA simply as a less detailed and less structured form of impact assessment. The result has often been criticism by decision-makers of the uncertain results emerging from SEA owing, in part, to the unverifiable nature of the approach and methods used (Noble et al. 2012). Part of the issue is that, in an attempt to be flexible, the range of methods and approaches used in SEA practice and recommended in guidance is restrictive and limited to a number of common, qualitative-based methods with more analytical-based and quantitative approaches underutilized and under-promoted (Noble et al. 2012). Quantitative-based approaches to SEA have been criticized for leading to ‘fictitious precision’ owing to the ‘fuzziness’ of PPP issues (Sommer 2005, p. 60). Arguably, structured and systematic methodology characterized by quantitative design does not diminish SEA's ability to be flexible or sensitive to context; and there are quantitative approaches to capturing the fuzziness of PPP issues and ensuring a more systematic and verifiable approach to assessment and decision-making. Such approaches have received relatively little attention in the SEA literature and there are few examples demonstrating how a systematic and quantitative SEA design can be sensitive to context and to the fuzziness of PPP issues – particularly sustainability, which itself has proven difficult to operationalize (see Brunner & Starkl 2004; Bina 2007).

In this paper we demonstrate a structured, quantitative approach to SEA in the context of the electricity sector planning in Saskatchewan, Canada. The purpose is to present a structured SEA methodology for addressing complex and often fuzzy PPP issues, with the aim of operationalizing sustainability considerations in the assessment process. The transportation and land use planning sectors have amassed substantial SEA knowledge (see Dalal-Clayton & Sadler 2005; Marshall & Fischer 2006), but SEA has yet to be applied on a similar scale in the electricity sector (Jay 2010). Jay and Marshall (2005) cite several concerns with SEA in the electricity sector, including the limited scope of application (supply and conservation, rather than networks), as well as application on too broad (at a policy level) or too narrow (specific energy sectors) a scale to be useful. As Schenler et al. (2002, p. 8) note, there is a demonstrated need for a ‘planning methodology framework that will assist decision-making process consistently with the long-term sustainable development of the electricity sector’. In the sections that follow the context for the SEA application is introduced, followed by the SEA design, methods and results. Attention then turns to lessons emerging for electricity sector planning and SEA methodology more broadly.

Electricity sector planning in Saskatchewan, Canada

Electricity generation and distribution in Canada fall under provincial jurisdiction. In each province the majority of electrical generation, transmission and distribution are provided by a single, provincially owned corporation. There are also smaller utilities, mostly municipally owned, which purchase power from the province, and several private sector and self-use industrial generating plants (Richards et al. 2012). In Saskatchewan, electricity is supplied by the Saskatchewan Power Corporation (SaskPower), a provincial crown corporation.

Saskatchewan, a western prairie province, is about 650,000 km², with a population of just over 1 million. Between 2006 and 2011 the province experienced an unprecedented 6.7% population increase, mainly owing to growth in the large-scale industrial and commercial sectors. Saskatchewan's economy is based primarily on the agricultural, mining (uranium, potash, coal), and petroleum and natural gas sectors, and has led Canada in growth of real GDP per capita in recent years (Richards et al. 2012). Saskatchewan's net electricity generation capacity for 2009 was 3840 MW, including 43.8% conventional coal, 29.5% natural gas (including 21.2% single cycle and 8.3% combined cycle natural gas), 22.2% hydro and 4.5% wind (SaskPower 2010). Provincial electricity demand has grown, on average, 1.3% per year over the past 10 years and is expected to grow by up to 3% per year in the next decade (SaskPower 2010). Increased demand will require a projected additional 4100 MW of generation capacity by 2030, which is greater than the total generation capacity in 2009 (SaskPower 2010).

There is a significant need for long-term strategic planning and assessment to guide the development of Saskatchewan's electricity sector (see Bigland-Pritchard & Prebble 2010). However, there is no formal SEA system in Saskatchewan and no strategic framework to guide the development and evaluation of alternative electricity production options. Environmental assessment in Saskatchewan is project-based under The Saskatchewan Environmental Assessment Act. There is some provision under the Act for the environmental assessment of plans, but this provision is limited to the forestry sector (see Gachechiladze et al. 2009).

Strategic environmental assessment framework

The sections that follow present the SEA framework and methods developed and applied to the Saskatchewan electricity sector. The SEA was undertaken by the authors as part of a research programme to provide guidance to electricity planning in Saskatchewan, and not as a formal initiative of the provincial government. The overall assessment framework was informed by conceptual guidance on SEA methodology (e.g. Gunn & Noble 2009; Croal et al. 2012), drawing also on analytical and decision support tools for SEA application (e.g. Noble & Storey 2001; Schetke et al. 2012). After establishing the context of and need for SEA, the framework consisted of the following:

1.	identifying SEA participants;
2.	developing assessment criteria;
3.	identifying PPP alternatives;
4.	assessing alternatives against the criteria;
5.	examining potential tradeoffs and identifying a preferred alternative(s); and
6.	determining the sensitivity of the assessment results to uncertainties and changing PPP conditions.

Expert-based assessment panel

An expert panel was assembled for the assessment; a combination of expert knowledge and practical experience is the typical approach to SEA application (Bao et al. 2004). Potential participants were sampled from stakeholders with expertise in electricity planning, energy development or environmental assessment, including environmental non-government organizations (E-NGOs), provincial energy crown corporations, regulators, industry and environmental consulting organizations. A few initial informants were contacted and asked to identify other potential participants. A total of 173 individuals were invited, of whom 44 people (25.4%) agreed to participate: 17 (38.6%) from government, including municipal and provincial ministries involved in electricity planning and environmental assessment; 15 (34.1%) from the private sector, including business and industry; and 12 (27.3%) E-NGOs.

Assessment criteria

The criteria against which to assess electricity alternatives were identified before the development of alternatives, ensuring that criteria selection was not biased by the alternatives (see van Huylenbrock & Coppens 1995). A preliminary list of criteria was derived from a review of recent plans and assessments in the international electricity sector, so as to ensure context-appropriate criteria (see Martensson et al. 2006; Public Service Commission of Wisconsin 2007; Ontario Power Authority 2008; Offshore Energy Environmental Research Association 2008; Department of Energy and Climate Change 2009; Partidario et al. 2010), and drawing also on the impact assessment and sustainability literature in energy sector planning (see Kowalski et al. 2009; Wang et al. 2009; Jay 2010; LaRovere et al. 2010). The preliminary criteria were reviewed by a subset of the expert panel, who were asked whether criteria were missing and, if so, to include them in the list and then rank the criteria based on importance for consideration in electricity planning. Responses were compiled into a final set of criteria (see Table 1) that attempted to operationalize a number of high-level sustainability principles (see Gibson 2006) within the context of applied SEA for electricity sector planning, namely: inter- and intra-generational equity (e.g. C5, C8); resource maintenance and efficiency (e.g. C6); socio-ecological system integrity (e.g. C2, C4); livelihood and sufficiency of opportunity (e.g. C3, C7); precaution and adaptation (e.g. C1); and socio-ecological civility and governance (e.g. C7).

Table 1 Assessment criteria for the electricity sector.

Criteria	Descriptions
C1: Adaptive capacity	Maximizes the ability to accommodate projected, as well as unanticipated future demand growth
C2: Emissions management	Minimizes emissions to air and water during electricity production, distribution and use over the life cycle of the system
C3: Employment and income sufficiency	Maximizes short- and long-term income and employment opportunities
C4: Ecological integrity	Ensures biodiversity conservation and ecological resiliency by minimizing use and disturbance of land and water resources
C5: Security of supply	Ensures secure and affordable access to energy supply for current and future generations
C6: Energy production and transmission efficiency	Meets electricity demands while minimizing energy use, raw material use and generation of waste during production and energy loss during transmission
C7: Aboriginal rights	Minimizes infringement on culture, traditional land use practices and Treaty Rights
C8: Public health and safety	Minimizes risk to public health and safety during electricity production and transmission

Electricity alternatives

Five policy-level electricity alternatives were developed, each describing an electricity mix for Saskatchewan over the next 30 years (Figure 1). An energy futures focus group of five individuals assisted in the development of scenarios. Focus group members were identified from among energy experts and provincial energy crown corporations based on their expertise and knowledge of the electricity sector. Draft scenarios were developed based on an analysis of SaskPower's assessment of electricity supply and demand in the province (SaskPower 2010), and presented to the group for comment. The five alternatives are as follows:

Figure 1 Resource mix for the current electricity regime in Saskatchewan and five alternative scenarios. (Conv. Coal, conventional coal; CCS Coal, carbon capture and storage coal; Small Scale, small scale on-site renewable electricity.)

•	Alternative 1 (A1) – a continuation of the current electricity production mix over the next 30 years. This could occur if there is a lack of substantial climate change policy, as well as limited research and development of new and renewable technologies. New conventional coal, single and combined cycle natural gas, hydro and wind facilities would probably be built.
•	Alternative 2 (A2) – an increased focus on nuclear electricity production, while still including other traditional means of production. This could occur if climate change policy were to be adopted that restricts or places heavy penalties on carbon emissions from coal-produced electricity. New small-scale nuclear, combined cycle natural gas, hydro and wind facilities would probably be built. No new conventional coal facilities or single-cycle natural gas facilities would be built. Several new small-scale nuclear power units with capacity no larger than that of current coal facilities (300–500 MW) would probably be built. It is assumed that reactors are built in areas with a sufficient workforce, access to cooling water and access to power markets.
•	Alternative 3 (A3) – an increased focus on renewables, including run-of-river hydro, reservoir hydro, biomass and wind, and small-scale on-site renewables. This could occur if climate change policy were to be adopted that restricts or places heavy penalties on carbon emissions from coal-produced electricity. New single and combined-cycle natural gas, hydro, biomass and wind facilities would probably be built. No new conventional coal facilities would be built. Use of small-scale renewables including solar, wind, biomass and other industry-scale or community-scale renewable electricity generation increases demand for local transmission networks. Owing to cost, it is assumed that solar technologies can only be implemented on a limited scale, and large-scale biomass facilities will be feasible. Electricity from biomass and hydro projects provide an additional benefit of efficient near-site electricity in remote communities, resulting in reduced power losses from transmission from distant facilities. Electricity generated through renewable technologies is much more variable than other generation technologies and, as a result, has implications regarding the reliability of power supply. Reliance on wind and run-of-river hydro has the potential to decrease system reliability. This reduced reliability is offset with single cycle natural gas peaking facilities, an additional 10% in the electricity mix.
•	Alternative 4 (A4) – an increased focus on large scale carbon capture and storage (CCS) replacing the majority of conventional coal generated portion of the electricity mix, while still including other traditional means of electricity production. This could occur if climate change policy were to be adopted that restricts or places heavy penalties on carbon emissions from coal produced electricity. New CCS coal, combined-cycle natural gas, hydro and wind facilities would probably be built. No new conventional coal facilities or single cycle natural gas facilities would be built.
•	Alternative 5 (A5) – an increased focus on electricity produced from natural gas, while still including other traditional means of electricity production. This could occur if there is a lack of substantial climate change policy instituted in the country, as well as limited research and development of new and renewable technologies. New single and combined-cycle natural gas, conventional coal, hydro and wind facilities would probably be built.

All alternatives included the assumptions that: (1) current demand-side management programmes will continue; (2) peak load (demand) in the province will continue to grow from 1.3 to 3% per year, resulting in total electricity demand by 2040 ranging from 4720 to 7905 MW; (3) generating capacity will continue to grow from 1.3 to 3% per year, resulting in total electricity production by 2040 between 5660 and 9320 MW; (5) solar power is not suitable for large-scale generation owing to high cost and low capacity; and (5) the majority of electricity is generated and consumed within Saskatchewan (see SaskPower 2010; Table 2).

Table 2 Required generation capacity, emissions, and electricity costs in 2040 for alternatives A1–A5.

		Required generation capacity (MW)
	2009	A1	A2	A3	A4	A5
Conventional coal	1690	2524	294	1233	881	1702
Hydropower	845	1174	1174	1468	1174	1174
Wind	173	470	294	1174	587	352
CC natural gas	320	763	704	1057	704	1468
SC natural gas	813	939	763	1115	763	1174
Nuclear			2642
CCS coal					1761
Small scale				117
Biomass				294
GHG emissions (million tonnes CO2e/GWh)^a	17.0	19.6	7.3	11.5	8.0	15.7
Cost of electricity ($/kWh)	0.06	0.11	0.12	0.14	0.13	0.10
a GHG emissions were calculated for the year 2025. Notes: CCNG, combined cycle natural gas; SCNG, single cycle natural gas; CCS coal, carbon capture and storage coal.

Greenhouse gas (GHG) emissions were calculated based on the electricity mix for each alternative over a one year period for a generation capacity in 2025 of 4830 MW using averages presented by Bigland-Pritchard and Prebble (2010).1 For comparison purposes, GHG emissions for 2010 were approximately 17 million tonnes of CO₂ (SaskPower 2010), based on 18,862 GWh of total electricity supplied and emission intensity of 0.9 tonnes CO₂e/GWh. The cost of electricity was estimated based on capital costs, power operation and maintenance costs.2 The cost of electricity in 2009 was approximately 6 ¢/kWh. Natural gas prices have been variable in the past. This may increase the future cost of A5, which relies heavily on natural gas.

Assessment methods

Methods used in SEA have significant bearing on the nature and quality of information made available to support decision-making (Noble et al. 2012). To address the fuzzy nature of sustainability and impact assessment at the PPP level, the assessment adopted a multi-criteria analytical approach. Multi-criteria analysis is useful when problems involve multiple criteria and options, and when problems are complex and characterized by competing knowledge and values (Herath & Prato 2006; Kain & Söderberg 2008). Multi-criteria analysis ‘aims to improve decision making by making choices about conflicting or multiple objectives explicit, rational and efficient’ (Finnveden et al. 2003, p. 102).

The expert panel assessed the electricity alternatives utilizing Saaty's (1982) analytical hierarchy process (AHP) – a multi-criteria decision-aid for prioritizing alternatives using multiple criteria. The AHP uses a weighted sum method where weights are applied to criteria based on ratio scales derived from paired comparisons (Wang et al. 2009), thereby enabling ‘decision makers to structure a complex problem in the form of a simple hierarchy and to evaluate a large number of quantitative and qualitative factors in a systematic manner under multiple conflicting criteria’ (Lee et al. 2007, p. 2863). The approach allows for the management of complex knowledge in planning for sustainability, but also provides an explicit measure of inconsistency (i.e., a consistency ratio), or internal conflict, in an individual's assessment (Saaty 1982) as an indicator of the overall quality of the assessments (Noble 2004).

The AHP was administered using Expert Choice web-based Comparion Suite software. Participants were asked to indicate the relative importance of each criterion when making decisions about electricity futures (see Figure 2) by weighting the criteria, pairwise, using a nine-point reciprocal scale from 1 (criterion i and criterion j are of equal importance) to 9 (criterion i is much more important than criterion j) to 1/9 (criterion j is much more important that criterion i; see Saaty 1982). Participants were then asked to assess each of the five electricity alternatives on the basis of each criterion using the same AHP paired comparison approach (see Figure 3).

Figure 2 Illustration of pairwise criterion weighting for ‘adaptive capacity’ (C1) and ‘emissions management’ (C2) in Expert Choice, online version based on decision goal ‘identifying a preferred electricity future for Saskatchewan’.

Figure 3 Illustration of pairwise alternatives assessment (A1 and A2) on the basis of criterion C1 ‘adaptive capacity’ in Expert Choice, online version.

Data analysis

Results were analysed using multi-criteria analysis and exploratory statistics to determine an overall ranking of electricity production alternatives based on the set of criteria. An assessment matrix was developed with the pairwise comparison scores for each participant. Following Saaty (1980), eigenvectors were then calculated using Expert Choice version 11 software for each assessment matrix to derive the weight of each criterion and a score for each alternative on the basis of each criterion.

Results were aggregated for the panel and analysed using exploratory data analysis in IBM SPSS version 18 statistical software package. Exploratory data analysis is well suited to SEA applications where data are often limited, but where there is a need to systematically assess competing options across multiple criteria. Non-parametric statistics were used to confirm exploratory data analysis results. Preference scores for electricity production alternatives were weighted using normalized criterion weights, so as to account for the relative significance of each criterion and allow an overall assessment of electricity production alternatives. A concordance analysis was used to test the robustness of the ranking of electricity production alternatives derived from the AHP process (Equation (1)):

where c_ii′ is the concordance set ii′, w is the the weighted impact score, i is the alternative i, and j is the alternative j.

An index of similarity (S) (Equation (2)) was used to determine how similar the ordering of alternatives was between the concordance analysis and the AHP.

where d is the the number of times the paired comparisons of a particular order agrees with the paired comparison values in the concordance matrix; n is the number of observations. To determine the magnitude of the differences among the ranking of alternatives, an interval ranking was performed based on Euclidean distance (Equation (3)).

where i _min and i _max represent the minimum and maximum values of the concordance sets, respectively.

Sensitivity analysis

Before a preferred option is identified some form of ‘sensitivity analysis’ is needed (Noble & Storey 2001). Sensitivity analysis allows the SEA analyst to examine the implications of the fuzziness of strategic-level decisions, the uncertainties associated with changing PPP conditions, and the subjective judgements and inconsistencies of SEA participants. First, a sensitivity analysis was performed to address inconsistencies in the assessment of alternatives, as well as uncertainties in the assignment of criterion weights. Inconsistent responses could originate from a participant's lack of understanding of the problem, uncertainty in assigning assessment scores owing to the complexity of the problem, incomplete information or intentional misrepresentation (Noble 2004). The sensitivity analysis in this case involved removal of inconsistent responses to determine if the ranking of electricity production alternatives significantly changed.

Second, sensitivity tests were performed to assess the robustness of the final ranking of alternatives against changing PPP conditions, as represented by changes in criteria weights under a series of ‘what if’ scenarios. The first sensitivity test (S1) examined the extent to which the final ranking of alternatives was contingent on continued economic growth in the province. The weight of C3 was increased to reflect an increase in the importance of ensuring employment and income sufficiency during a period of economic stagnation, and the weight of C2 decreased to indicate a trade-off of environmental standards. The second sensitivity test (S2) examined the impact of an increase in the weight of C7, Aboriginal rights, in a scenario where electricity development or distribution in the province was contingent upon access to Aboriginal lands or settlement of Treaty rights. The third sensitivity test (S3) examined the impact of a scenario where recent international nuclear incidents resulted in increased concerns over public health and safety (C8) and emissions management (C2) in electricity production.

Assessment results

Criteria weights

The median weights for the assessment criteria are presented in Table 3. Health and safety (C8) and security of supply (C5) were the most important criteria with respect to electricity development, closely followed by ecological integrity (C4) and energy production and transmission efficiency (C6). Aboriginal rights (C7) was identified as the least important criterion, followed by employment and income sufficiency (C3). Based on the 95% confidence intervals for the median, the ranking of criteria for consideration in electricity sector planning in Saskatchewan was as follows: C8>C5 I C4 I C6 I C2 I C1>C3 I C7, where ‘>’ indicates a significant difference between criteria based on median weights, and ‘I’ indicates a lack of difference.

Table 3 Criterion weights and 95% confidence interval for the median.

Criteria	Median	95% CI
C1 – Adaptive capacity	0.0995	± 0.0224
C2 – Emissions management	0.1110	± 0.0336
C3 – Employment and income sufficiency	0.0410	± 0.0105
C4 – Ecological integrity	0.1280	± 0.0277
C5 – Security of supply	0.1320	± 0.0326
C6 – Energy production and transmission efficiency	0.1210	± 0.0174
C7 – Aboriginal rights	0.0400	± 0.0135
C8 – Public health and safety	0.1920	± 0.0351
95% CI, 95% Confidence interval for the median; n = 44 participants.

Electricity production preference scores

The median preference scores and 95% confidence intervals for electricity alternatives, weighted based on the criterion weights (Table 3), are shown in Table 4. Alternative A3, the renewables-focused alternative, was the preferred alternative, followed by A5 and A2, the natural gas and the nuclear-focused alternatives, which were scored at 0.127 and 0.125, respectively. Alternative A4, the carbon capture and storage option, was scored at 0.113. The least preferred was A1, a continuation of the current electricity mix, scored at 0.069. Based on a Wilcoxon test for difference, the panel's ranking for the province was as follows: A3>A5 I A2 I A4>A1, where ‘>’ indicates a significant difference between alternatives based on weighted assessment scores, and ‘I’ indicates a lack of difference.

Table 4 Weighted aggregate preference matrix*.

	A1 W^a		A2 W		A3 W		A4 W		A5 W
Criteria	Median	95% CI	Median	95% CI	Median	95% CI	Median	95% CI	Median	95% CI	NW
C1	0.0064	± 0.0021	0.0157	± 0.0077	0.0244	± 0.0088	0.0088	± 0.0054	0.0165	± 0.0048	0.1151
C2	0.0062	± 0.0018	0.0239	± 0.0064	0.0455	± 0.0183	0.0193	± 0.0057	0.0128	± 0.0039	0.1284
C3	0.0026	± 0.0009	0.0070	± 0.0023	0.0123	± 0.0052	0.0052	± 0.0017	0.0047	± 0.0013	0.0474
C4	0.0068	± 0.0022	0.0163	± 0.0053	0.0307	± 0.0179	0.0134	± 0.0044	0.0136	± 0.0064	0.1481
C5	0.0133	± 0.0074	0.0209	± 0.0072	0.0193	± 0.0075	0.0148	± 0.0073	0.0194	± 0.0080	0.1527
C6	0.0087	± 0.0017	0.0178	± 0.0066	0.0384	± 0.0118	0.0130	± 0.0035	0.0186	± 0.0054	0.1400
C7	0.0074	± 0.0022	0.0047	± 0.0026	0.0081	± 0.0034	0.0094	± 0.0039	0.0085	± 0.0025	0.0463
C8	0.0178	± 0.0048	0.0187	± 0.0117	0.0638	± 0.0196	0.0287	± 0.0088	0.0331	± 0.0094	0.2221
Sum	0.06933		0.12503		0.24256		0.11263		0.12722		1.0000
a A1thinsp;W, ‘Weighted’ alternative based on median criteria weights; NW, normalized criteria weights; 95% CI, 95% confidence interval for the median; n = 44 participants.

Results by participant group

Criterion weights by group

Median criterion weights calculated by group are shown in Table 5. The 95% confidence interval indicates that, for government participants and E-NGOs, criteria weights were not statistically different. For the private sector, criteria C2, C8, C4, C6, C5 and C1 were not statistically different, but they were weighted more heavily than C3 and C7, which were also not different from each other.

Table 5 Median criterion weights and 95% confidence intervals by group.

Criteria	Government	95% CI	Private sector	95% CI	E-NGOs	95% CI
C1	0.101	± 0.053	0.090	± 0.028	0.120	± 0.079
C2	0.093	± 0.025	0.193	± 0.074	0.107	± 0.051
C3	0.030	± 0.020	0.033	± 0.015	0.047	± 0.018
C4	0.072	± 0.046	0.132	± 0.050	0.150	± 0.091
C5	0.182	± 0.069	0.101	± 0.047	0.109	± 0.048
C6	0.114	± 0.017	0.114	± 0.034	0.138	± 0.038
C7	0.050	± 0.041	0.024	± 0.023	0.037	± 0.013
C8^a	0.193	± 0.050	0.191	± 0.055	0.194	± 0.094
a Differences between criteria confirmed by Tamhane post-hoc test for significance value ≤  0.05.

Using normalized criteria weights (Table 5), the aggregate ranking of alternatives is A3>A2>A4>A5>A1. Weighted alternative rankings, using normalized criteria weights, were determined for each of the participant groups (Figure 4, Table 6). For the aggregate panel, and for all sub-groups, alternatives A2, A4 and A5 were not statistically different. Based on participant group, government, private sector and E-NGOs consistently showed a demonstrated preference for A3, with A1 consistently the least preferred. This is consistent with the results of the aggregate panel, suggesting that the aggregate assessment scores captured the preferences of each of the stakeholder groups.

Figure 4 Weighted alternative preference scores by group.

Table 6 Alternative rankings.

Participant group	Alternative preference
All participants	A3>A2 I A4 I A5>A1
Government	A3>A5 I A2 I A4>A1
Private sector	A3>A2 I A5 I A4>A1
E-NGOs	A3>A2 I A5 I A4>A1
‘>’ Indicates significant difference between alternatives, based on Wilcoxon test, p ≤  0.05; I indicates no statistically significant difference between alternatives, based on Wilcoxon test, p ≤  0.05.

Alternatives A2, A4, A5, however, for the aggregate panel and for the sub-groups, differ in terms of rankings. Even though these alternatives are considered to be not statistically different, the aggregate assessment scores do not capture the preferences of the stakeholder groups. This indicates that there are differences between the aggregate panel and the stakeholder groups, as well as among the stakeholder groups themselves, regarding alternative preferences.

Robustness of assessment results

Concordance analysis

Results of the concordance analysis (Table 7) on the normalized weighted aggregate preference scores (see Table 5) confirmed the AHP ranking of electricity alternatives. An interval ranking of alternatives, based on the scaled concordance results and Euclidean distance, is shown in Figure 5. Results indicate that A3, the renewables alternative, was consistently the most preferred. For the aggregate panel, A3 was three times as preferred as the next alternative, A5, the natural gas alternative. The private sector was similar to the aggregate panel; however, for the private sector A3 was more than six times preferred to A5. The government and E-NGO groups similarly identified a strong preference for A3; however, there were more competing alternatives for each of these groups with respect to A4, A2 and A5. For the government group, A3 was more than three times as preferred compared with A4; for the E-NGOs, A3 was only 1.5 times more preferred than A4. Across all groups, A1, the status quo, was consistently the least preferred.

Table 7 Concordance matrix and AHP matrix for alternative preference using normalized criteria weights.

Group	Concordance matrix^a	AHP matrix^a
Aggregate panel	A3>A2>A5>A4>A1	A3>A2>A5>A4>A1
Government	A3>A5>A2>A4>A1	A3>A5>A2>A4>A1
Private sector	A3>A2>A5>A4>A1	A3>A2>A5>A4>A1
E-NGOs	A3>A2>A5>A4>A1	A3>A2>A5>A4>A1
a ^a Index of similarity = 1.0 for the concordance analysis.

Figure 5 Scaled alternative preference scores. Gov't, Government; E-NGOs, environmental non-government groups.

Sensitivity analysis

A sensitivity analysis was performed on the consistency ratio (CR), derived from the AHP, as well as changes to criteria weights. The CR of an assessment matrix is a measure of how the assessment scores compare with a random matrix; a normal consistency ratio is considered to be 0.1 (Saaty 1980). The mean consistency ratio of all responses was 0.142; approximately 18.5% of responses had a consistency ratio of 0.2 or greater. Hence, 0.2 was chosen as an acceptable level of inconsistency. Rather than the number of inconsistent responses, what is important is how the inconsistencies affect the decision results. The aggregate assessment matrix was recalculated from the AHP results with individual responses with a CR>0.2 removed. Results showed a ranking of electricity production alternatives based on those responses with a CR < 0.2 as A3>A5>A2>A4>A1, which is not different than the aggregate AHP results in Table 4. Inconsistent responses, which may be due to the complexity of the problem at hand or a lack of detailed information on the alternatives, had no significant influence on the overall results.

A set of three sensitivity tests was used to see if changes in criteria importance affected the alternative rankings (Figure 6). In S1, with a 95% increase in C3, employment and income, the ranking remained unchanged, indicating that the ranking is robust against significant changes in priorities regarding employment and income sufficiency. In S2, a 95% increase in C7, Aboriginal rights, resulted in a change to the ranking such that A3, the renewables alternative, was no longer preferred. This indicated that, if Aboriginal rights were to become more important in terms of access to lands for electricity production or distribution, then A5, the natural gas alternative, would out-compete A3. Relative preference for the status quo (A1) would increase slightly, and A2 would be the least preferred. In S3, with an 80% increase in C8, public health and safety, the overall ranking again remained unchanged, indicating a robust ranking.

Figure 6 Scaled alternative preference scores for the aggregate panel and sensitivity tests 1–3.

Discussion

Implications for electricity sector development in Saskatchewan

The preferred development path for the province was A3, a renewables focused future; the least preferred was A1, a continuation of the current trajectory. However, a number of factors and implications must be taken into account when implementing the preferred alternative. According to Cherp et al. (2007, p. 633), ‘external factors and/or internal organizational dynamics’ that are critical to the successful implementation of the PPP must be identified and considered, including the feasibility of implementation and whether a supportive institutional environment exists for the preferred alternative (Gunn & Noble 2009), along with tradeoffs associated with the preferred PPP. In the Saskatchewan case, there are several implications for implementation, including the economic viability of renewables in the short term owing to infrastructure requirements, increased cost of electricity, environmental impacts from the development of hydropower and shifting priorities regarding Aboriginal rights.

A renewables focused future will require large-scale development of numerous power generation facilities (see Table 2). These additional infrastructure needs will require significant investment to ensure that sufficient capacity is available under a renewables alternative. As a result, as well as costs due to operations and maintenance, the cost of electricity is projected to increase. Alternative A3 had the highest associated cost of electricity, at approximately 14 ¢/kWh, which is slightly higher than the 11 ¢/kWh cost of A1 under the current electricity mix (Table 2). While greenhouse gas emissions under A3 would decline to 11.5 million tonnes of CO₂/year, compared with the current mix (A1) producing 19.6 tonnes of CO₂/year (Table 2), there would probably still be significant environment implications associated with A3 given the inclusion of hydropower development. Given the strong preference for A3, but considering the infrastructure investment and the time required to undertake such a significant policy shift toward a renewables option, the ranking of A1 may have been different if it was accompanied by the sustainable alternative of a more ambitious demand-side management programme.

Results of the sensitivity tests also highlight potential future implications for the renewables option. Sensitivity test S2, for example, showed that the only criterion that would shift the preference away from A3 would be an increase in the importance or relevance of Aboriginal rights (C7) to decisions that concern electricity development and distribution. If Aboriginal rights become the dominant criterion in making decisions about future electricity development, then the most preferred alternative would shift to A5, a natural gas scenario, and A2, the nuclear scenario, would become least preferred. If Aboriginal rights become a priority in electricity sector planning, or if development requires access to or settlement of issues that relate to Treaty lands, there are significant political issues to resolve so as to ensure the viability of a renewable focused future.

Opportunities for advancing SEA methodology

Strategic environmental assessment has often been criticized for adopting a vague and inconsistent approach, for the uncertainty of assessment results, and the unverifiable nature of the methods used (see Auditor General 2004; Liou et al. 2006; Noble et al. 2012). The aim to ensure flexibility and sensitivity to context, however, does not need to trade-off structure and consistency in SEA guidance and methodology. Good SEA both provides the needed methodological structure for practitioners to rely on in applying SEA, so as to ensure replicability and confidence in the process and results, and allows for flexibility in the scope of alternatives, the choice of assessment methods, the nature of the criteria developed and the scope of participation.

In this paper, a systematic and structured SEA framework was operationalized using a quantitative design. Although quantitative-based approaches to SEA have been dismissed as inappropriate (see Sommer 2005), a quantitative approach can be used effectively to address the uncertainty and fuzziness around strategic-level decisions (see Brunner & Starkl 2004; Schetke et al. 2012). The SEA practitioner is able to repeat the SEA analysis under different ‘what if’ scenarios and generate reliable results without having to collect new assessment data. This provides flexibility for the practitioner in examining the robustness of the recommended PPP. In the Saskatchewan case, we demonstrated effects on the preferred PPP of uncertainties in participant assessment scores, as measured by a consistency ratio, and the effects of changing priorities, as measured by changes in the relative criterion importance of employment and income self-sufficiency, Aboriginal rights, and public health and safety. Using this approach there is no limit to the number of sensitivity analyses that could be undertaken to examine how uncertain futures and changing organizational priorities or other, external factors (see Cherp et al. 2007) may affect the preferred option. This provides important information for decision-makers, and allows them to understand the potential political risks associated with certain strategic choices under uncertain future conditions. The means by which the preferred PPP is identified is transparent and the process can be replicated.

There is also an opportunity in the framework to extend the scope and level of engagement in SEA, with minimal effort, beyond what may be possible using less structured approaches. In our example we included a relatively small sample of experts; however, the on-line assessment tool could easily be extended beyond the expert panel to include members of the public from across the province. This would allow the SEA practitioner to identify potential spatial variations in PPP preferences, or to examine PPP preferences based on participant affiliation – such as Aboriginal groups, environmental organizations or electricity providers. Using a sensitivity analysis, results can then be examined for sensitivity to stakeholder preferences and the output can be traced backwards to determine the relative influence of participant groups and the weighting of assessment criteria on the preferred option. This is important information for SEA decision-makers in understanding the level of dissent or consensus amongst the various groups involved in the SEA process (see Noble 2004).

Finally, the case application demonstrated one approach to operationalizing sustainability principles in SEA, specifically in the context of the electricity sector and through the use of assessment criteria. Marsden (2002) argues that SEA can play a role in sustainability if ‘simple (and) pragmatic’ indicators are used; however, we argue that sustainability in SEA has been far from pragmatic. Sustainability is mentioned as an overarching goal in many SEAs, and there are some well-recognized sustainability principles (see Gibson 2006), but rarely does this translate to direct assessment application (see White & Noble 2012). There have been few concrete examples and little guidance as to how to operationalize sustainability principles in an applied SEA context (Retief 2007). The Saskatchewan case demonstrated one approach to developing operable criteria that capture general sustainability principles. Of course, one option to increase sustainability in SEA is to adopt only ‘strongly sustainable’ alternatives from the outset; however, this may not be realistic in all SEA contexts.

Conclusion

This paper demonstrated an expert-based, quantitative SEA process to assess alternative futures for electricity development based on a set of defined criteria. The proposed SEA framework allows for replicability and sensitivity testing and provides credibility and transparency in the assessment methodology, but it also allows for flexibility in participation, the range of alternatives, and how criteria are designed to accommodate sustainability principles that are sensitive to the context of the electricity sector. The case demonstrated structure in SEA design and a quantitative approach to address the uncertainty and fuzziness that often surrounds PPP assessment and sustainability. That being said, the benefits of structured and quantitative approaches to SEA are under-reported and under-promoted in SEA practice (Noble et al. 2012), and there is currently only limited awareness regarding quantitative SEA designs. More attention is needed on reporting the lessons emerging from SEA applications with quantitative designs, and on developing methodological guidance to aid in the choice of appropriate assessment techniques. Although further theoretical development of SEA is still needed, additional reporting of the lessons learned from SEA case application are important to advance SEA design and build better frameworks to guide effective SEA for sustainability.

Acknowledgements

This research was supported, in part, by the Social Sciences and Humanities Research Council of Canada.

Notes

^1. Load factors are based on information provided by SaskPower (conventional coal, wind), the project's advisory committee (single cycle, combined cycle natural gas, hydro, CCS coal) and Graham et al. (2005) (nuclear, small scale, biomass). Load factor is the ratio of average load (intensity of usage) to generation capacity in a period, or a measure of the actual output of a power plant compared with its maximum theoretical output. Small scale GHG emission rates and load factors were based on solar photovoltaics. An additional 10% of single cycle natural gas is included in A3. It was assumed that single cycle and combined cycle natural gas facilities have the same GHG emission intensity.

^2. Capital and power costs are from SaskPower. Power costs include load factors and the impact of a cost of carbon on GHG emissions and sales revenue for CO₂ captured in A4. Power cost for a simple cycle option is not applicable as it typically has low capacity factors owing to its peaking operation. Power cost for nuclear includes an allowance for decommissioning and interim fuel storage in A2.