Determining sustainability impact assessment indicators

Abstract

Sustainability impact assessment (SIA) can enlighten the decision-making process for major public investment projects. This paper addresses the use of SIA on projects, and suggests that since policies constrain plans which constrain projects, it is necessary to assign SIA indicators to the strategic, tactical and operational levels. We have examined (1) what indicators are actually chosen as being the most significant and (2) how assessors should choose their indicators. We let 538 of our students – assembled in 121 groups – identify indicators for an SIA of a major road project, and analysed the propensities of an exemplar SIA. We observed a lack of predictability in the students' choice of indicators, and that the exemplar SIA bore similar characteristics. Predictability in a tool designed to assure enlightened decision-making is essential, since lack of it can entail challenges concerning – just to mention some – transparency issues, governance, stakeholder participation and effectiveness. Discussions related to choice of indicators for SIA concern their commensurability and whether it is possible to price them or not. We point out that first step should be to make sure all relevant indicators are considered. Afterwards we propose a framework with nine different categories that will help assessors to identify an exhaustive indicator list. Our main argument is that indicators should be assigned to the strategic, tactical or operational level, as well as to one of the economic, social or environmental dimensions.

Keywords::

• indicator
• sustainability
• assessment
• strategic
• tactical
• operational

1. Introduction

The Organisation for Economic Co-operation and Development (OECD) has introduced the concept of sustainability into the domain of project management (OECD 1991). Sustainability is, together with efficiency, effectiveness, impact and relevance, a success criterion according to which projects are to be assessed. Recent years have witnessed the creation of a considerable number of impact assessment tools pertaining to sustainability assessment (Haavaldsen et al. 2013). We find the most interesting of these analytic tools to be sustainability impact assessment (SIA),1 as described in its latest version by the OECD (2010). We are familiar with the definition used by Bond et al. (2012, p. 60), stating that ‘Sustainability assessment can be any process that directs decision-making towards sustainability’, and that ‘this definition encompasses many potential forms of decision making from choices of individuals in everyday life through to projects, plans, programmes or policies more familiarly addressed in the field of impact assessment’. We also noted Morrison-Saunders et al.'s (2014, p. 7) conclusion that ‘The current state of practice suggests that we as a community are not sufficiently unified at present, nor clear enough on the common purpose of impact assessment, nor the value-added to decision makers’. So, in sum there is a need for improving the current state of practice.

The idea to write this paper came to us after presenting the concept of SIA as described by OECD to our engineering students for three years in a row. After the lectures, we gave them group assignments where they had to carry out an SIA of an actual major road project. The students carried out surprisingly good assessments, with relevant analyses.

An SIA cannot overlook that an investment project is initiated to support a larger plan, programme or policy. Hjelmbrekke et al. (2014) found that ‘an alignment of project outputs with strategic goals happens to a certain extent in some projects, but it seems random and very dependent on individuals assuming broad responsibility’. The strategy (i.e. overall policy of the financing party) must be supported by the activities at operational level (among other activities, investment projects). Policies constrain plans which constrain projects, and separate SIAs are typically conducted at these different levels. The assessment indicators should be adapted to the actual level. When selecting indicators for an SIA of a project, the assessor must have in mind that the project is initiated in order to support an overall policy. For instance, it seemed to be natural for all our students to assess project investment costs (in the operational perspective), while neglecting to consider the total life cycle costs (in the strategic perspective). After this experience, we formulated two research questions:

• What indicators are actually chosen as the most significant?

• How should assessors choose their indicators?

Our interest lies in examining SIA from what we consider to be a seminal perspective. We intend to highlight – what in our opinion is – one of the fundamental challenges to the use of SIA on project level, notably the question of how to achieve a balanced assessment, without ignoring important impacts. An SIA of a project must assess if the project is in line with policies at strategic level, for example at a national – or supranational, for that matter – level. As Macrae (2008, p. 84) comments, when using assessment tools, ‘[m]uch depends on the set questions […] built into the tool’.

The number of theoretical indicators can be limitless, and the eager evaluator might drown in information and concerns. There are three other challenges to the use of the indicators in SIA related to weighting, measuring and pricing. First, the weight given to the different indicators and classes of indicators (economic, social and environmental) is not completely objective, since it comes from the assessor (Hofstetter 2002; Schmidt & Sullivan 2002a, 2002b). Is the negative impact on local merchants' commerce a weighty argument in regional investment projects aiming at reducing traffic through urban landscapes? Is the single nest of a rare bird enough to stop the expansion of a transregional highway? Second, significant research literature exists concerning the application of specific methods applied to analyse and measure the impacts within the three dimensions, respectively. The choice of methodology for measuring ‘air quality’, for instance, is by no means value neutral, not to mention the difficulty in measuring popular but intrinsically complex social indicators such as ‘quality of life or well-being’ (Bell & Morse 2008, p. 18). Third, literature has shown how the economic consequences of a given project have the advantage of being quantifiable in monetary values, and can therefore overshadow other concerns which cannot be quantified directly (Heinzerling & Ackerman 2002). The social and environmental impacts are often more difficult to quantify, and consequently they are prone to be omitted. Our research questions deal with the challenge of choosing indicators that will give a balanced assessment, and not with weighting, measuring or pricing of indicators.

2. Theoretical framework of our analysis

2.1. Sustainability

A fundamental insight is that the economy, the society and the environment are interdependent. Rather than seeing them as being discrete, well-separated entities, the economy is understood as existing within a society and unable to develop in the long term without taking society's needs into account. The society similarly exists within the boundaries of the environment; if environmental concerns are not addressed in a proper manner, society itself cannot exist in the long term. On the basis of such comprehension, it becomes clear that an assessment of the sustainability of any investment project ought to assess its impacts within all the three dimensions illustrated in Figure 1.2

Figure 1 The three dimensions of sustainability. A common understanding is that sustainability can be assessed in economic, societal and environmental dimensions.

It is well known that the concept of sustainability is multifaceted, and that its use varies largely within different contexts. Gomis et al. (2011, pp. 173–174) refer specifically to the idea of ‘sustainable development’ when using the term ‘sustainability’, and point out that ‘sustainable business’, ‘sustainable technology’, ‘sustainable agriculture’, ‘sustainable economics’, etc., are all buzzwords of the literature today. In fact, the understanding of the concept varies even at the individual level; ‘the precise meaning of sustainable, and what it embraces, varies depending upon who is using it and in what context’ (Bell & Morse 2008, p. 5). The terms sustainability and sustainable development can have different meanings (Lele 1991), but we do not want to make a major point out of this. Therefore, in the following, we have used them synonymously.

According to Adams (2006), ‘[a]nalysts agree that one reason for the widespread acceptance of the idea of sustainable development is precisely [its] looseness. It can be used to cover very divergent ideas [….]. The concept is holistic, attractive, elastic but imprecise’ (p. 3). Indeed, as Bond and Morrison-Saunders (2011) point out, the premises onto which these definitions are based prove to be ‘subject to value judgments’ and are thereby to a certain extent relative, since ‘there are likely to be differing expectations of the goals of the assessment’ (p. 2). This differentiated use has in fact, according to Marshall and Toffel (2005), ‘nearly rendered the term sustainability meaningless’ (p. 673), at least with regard to being understood as a precise term. Bell and Morse (2008, p. 10) have also noticed that this can be a source of much frustration:

Almost every article, paper or book on sustainability bemoans the fact that the concept is broad and lacks a broad consensus; this is usually followed by the authors' own preferred definitions, which in turn add to the lack of consensus!

This frustration does not mean, however, that assessing the sustainability of large investment projects is valueless. As Rahimifard and Clegg (2008) comment, the novel focus on sustainability creates ‘tremendous opportunities […] for the engineering community to develop methods, technologies and tools which can transform the realisation of sustainable development principles from a conceptual vision to a common reality’ (p. 1). As a response, the ‘impact assessment is an emerging scientific field that includes a variety of tools and methods and that serves various activity and decision making levels’ (Helming et al. 2008, p. 1). In fact, as Bell and Morse (2008, p. 3) comment, the ‘idea of using indicators as a means of gauging sustainability has become extremely popular’.

In order to provide clarity concerning what is assessed, assessment tools such as SIA depend on firm conceptual specification. The OECD (2010) lacks, in fact, any such clear-cut definition of sustainability. In our view, the most perspicuous among the plethora of definitions available is the one provided by the OECD within the context of development projects. According to this definition (OECD 2002, p. 36),3 sustainability within an investment project context is to be understood as:

The continuation of benefits from a development intervention after major development assistance has been completed. The probability of continued long-term benefits. The resilience to risk of the net benefits flows over time.

2.2. SIA

In its conceptual foundations, the SIA is a relatively simple tool. Its most significant feature is the collation of quantitative and qualitative data within the three pillars of economic, environmental and social sustainability. The OECD (2010, p. 4) describes the SIA as an ‘approach for exploring the combined economic, environmental and social impacts of a range of proposed policies, programmes, strategies and action plans’, underlining that such ‘assessments can also assist decision making and strategic planning throughout the entire policy cycle’. The proposed field of use for the tool is, in other words, extensive. In this paper, we limit the analysis to SIA usage in a narrower field, notably that of the front-end assessment of large public investment projects.4

The research literature describes a manifold use of the SIA procedures. Morrison-Saunders and Pope (2013), for instance, describe how sustainability assessments can be used in order to evaluate and manage trade-offs. They illustrate concessions that can be given between the economic, social and environmental spheres.

SIA is not intended to provide any final (technocratic) answer to investment project assessment; rather, it is a tool for exposing the most significant impacts of projects – positive as well as negative – thereby improving clarity to what Waldron (2012, p. 751) describes as the ‘painstaking complexity of policy analysis’ permitting for enlightened decision-making. Distilling the information available into comprehensible entities through the use of analytic tools proves in fact essential to enlightened decision-making in contemporary societies. As Lawrence (1997, p. 179) comments, ‘[i]n the developed world there is an overwhelming amount of data’; in fact, we often have ‘more data than we can ever use’ (p. 185). What is lacking, then, is not quantity of data, but ‘an understanding of what is important and the resolve to act’ (p. 185). In order to arrive at enlightened decision-making, data need to be ‘aggregated into a form that is both relevant to the issue at hand and accessible to those who are engaged in addressing it’ (p. 180).

This point is in fact essential, and viewed that at ‘strategic planning levels, we usually face uncertainties on impact predictions and we touch on value questions which cannot be solved solely by expert knowledge’ (OECD 2008, p. 38). On the basis of this, it is clear that ‘whatever the choice of method and methodology, an SIA is an aid to political decision-making, not a substitute for it’ (OECD 2010, p. 14).

As Bond and Morrison-Saunders (2011) further point out on the subject of adjacent methodologies (in their case sustainability assessment), such tools are inherently good, but we need to recognize that they are in their ‘formative years of development’, a period where ‘practice will be affected by a lack of familiarity amongst practitioners and a lack of capacity which is common when new tools are applied’ (p. 2). There in fact seems to be, as Hacking and Guthrie (2008) comment, ‘a great deal of work’ required in order to ‘develop assessment techniques that deliver practical results capable of supporting the lavish policy-level commitments to SD [sustainable development]’ (p. 86).

2.3. Sustainability indicators

A necessary element in the management of a system is developing knowledge of that system – identifying relationships and aggregating information to discern patterns (Bossel 1999). One approach to collecting, condensing and communicating this knowledge is through the use of indicators. An indicator represents a distillation or abstraction of reality – a system element or variable chosen for its ability to describe a specific characteristic in the state of a system (Meadows 1998; Mayer 2008) ‘that can be used to illustrate and communicate complex phenomena simply’ (European Environment Agency [EEA] 2005, p. 7).

The most well-known indicator frameworks in use for sustainability assessment and presentation are based on the Pressure-State -Response (PSR) framework developed by the OECD (OECD 1993; Segnestam 2002). Core OECD indicators include economic, social and environmental factors, from which relevant indicators can be chosen and reported (OECD 1993; Segnestam 2002). The PSR framework has since been expanded to better attempt to elucidate the web of causes and effects inherent to complex systems. Expanded PSR frameworks include:

• Pressure-State-Impact-Response (PSIR),

• Driving Force-Pressure-State-Impact-Response (DPSIR),

• Driving Force-Pressures-State-Exposure-Effects-Actions (DPSEEA).

The DPSIR framework has been utilized extensively by the EEA, as they maintain that it provides a ‘systems analysis view’ meant to expose the interplay between socio-economic activities and the environment (Smeets & Weterings 1999). In this framework, socio-economic development is seen as the driving force exerting pressure on the environment, causing a change in environmental state, resulting in impacts on ecosystems, human health and resources – inducing a societal response that can feed back on any of the elements or the linkages between them (Smeets & Weterings 1999). The EEA has developed a core set of indicators to support high-level European policy, not specific project-based assessments (EEA 2005, 2013).

The OECD has also put forth SIA as a specialized approach to augment more general assessments, which could involve PSR or other frameworks. SIA is distinctive in its focus on maintaining integration between the social, economic and environmental factors inherent to sustainability (OECD 2010).

In our view, the main attraction of SIA when compared to other analytic frameworks seems to be the focus on overall impacts of investments, thereby permitting a proper analysis. This analysis can be conducted at both the operational, tactical and strategic level.

2.4. Unpredictability can entail uncertainty

Lack of predictability in the choice of indicators in an SIA can entail challenges concerning – to mention some – transparency, governance, stakeholder participation and effectiveness. Transparency will in most respects be advantageous. Tennøy et al. (2006, p. 55) present evidence of ‘the uncertainty inherent in EIA [Environmental Impact Assessment] predictions, and a lack of both communication of such uncertainty and transparency in the prediction processes’. They suggest that improving prediction performance will reduce these disadvantages. Cashmore et al. (2009, p. 91) have concerns on the effectiveness of impact assessment instruments, ‘in terms of theoretical versus empirical emphasis, policy integration concerns, contributors’ beliefs and framing etc.'. Bond et al. (2012, p. 60) suggest that ‘sustainability assessment is currently in this initial phase of development, where early practice is being adapted to fit new situations and new contexts as practice has not yet reached a situation where particular methods or approaches are proven to work well’. They refer to Cherp (2001), and conclude that ‘further development is important because the imposition of assessment processes in contexts for which they were not designed has been found to be problematic in the past’ (Bond et al. 2012, p. 60). In sum, it can be said that including the significant impact indicators will make the outcome of an SIA more predictable and thereby more useful.

2.5. Insights from project management literature

Obviously, when carrying out an SIA, it is important to understand in which context the project ought to be assessed. There are two major insights from the project management literature that illustrate what we think should be emphasized.

The first of these insights is that projects – and correspondingly their impacts – can be analysed from different perspectives. The questions on why and how the project is carried out can be answered from different analytic perspectives. Based on de Wit (1988) and others, Cooke-Davies (2002) draws a fundamental distinction between project success and project management success. Project success is ‘measured against the overall objectives of the project’ (Cooke-Davies 2002, p. 185). The overall objectives will be assessed from the tactical and strategic perspective. Project management success is ‘measured against the widespread and traditional measures of performance against cost, time and quality’ (Cooke-Davies 2002, p. 185). These measures will be assessed from an operational perspective. Proper assessments of investment projects depend on an understanding of why and how the project is carried out. Samset (2010) writes that different stakeholders will have different perspectives and interests, and they often conclude differently in assessing the same project. The initiating party will have a strategic interest in what value the project add to business or to society. The end users will have a tactical interest in the results and services delivered from the project. The contractor will focus on the operational measures cost, time frame and produced quality.

Let us illustrate this point according to intended impacts of a road construction project. From an operational perspective, a road is assessed according to quality criteria as time, cost and quality (contractor). From a tactical perspective, a road can be examined according to its car traffic capacity (user). From a strategic perspective, what is of interest is the regional economic development following the increased traffic and/or reduced transportation time (initiating party).

The second insight is that these separate analytic perspectives are often characterized by different time horizons. Encyclopaedia Britannica (2013) put it in general terms, stating that the distinction between tactics and strategy depends on three considerations, where the first is ‘the longer the effect of a decision and the less reversible it is, the more strategic it is’. Shenhar and Dvir (2007, p. 26) relate more specifically to project management, and suggest that a ‘comprehensive assessment of project success in the short and the long term can be defined by five basic groups of measures’.

‘In the short term […] the project efficiency dimension is critical. In fact, it is the only one that can be measured in concrete terms at this stage. […] And finally, preparation for the future, which expresses the long-term benefits of the project, affects the organization only after years have passed. (Shenhar & Dvir 2007, pp. 29–30)

It is significant to remark when different impacts of investment projects appear. Operational impacts (noise, dust, etc., related to project efficiency) will typically appear within a shorter time frame than impacts relevant to a tactical perspective, while the corresponding strategic impacts (i.e. preparation for the future) take even longer time to unfold. It is also worth noticing that in theoretical literature – not only in practice – it can be difficult to make clear distinctions between the different success dimensions and different analytic perspectives.

2.6. Improving SIAs with insights from project management literature

Large investment projects entail significant impacts, not only intended ones defined by the project purpose and objectives, but also unintended ones. Consequently, unintended impacts need to be understood according to the same analytic perspectives. Let us illustrate this point by way of the same road example.

From an operational perspective, noise and emissions during the construction period are typical unintended impacts that need addressing. From a tactical perspective, the undermining of bus or railway companies can be considered a typical unintended impact. And from a strategic perspective, an increased possibility to access out-of-region shopping may harm local commerce, thereby annulling the anticipated economic benefit. Similar impacts may be identified within all three pillars of the sustainability perspective.

The point is that an impact can be characterized as sustainable from one perspective, but not sustainable from another perspective. It is essential to avoid confusing the perspectives when performing a sustainability assessment. Haavaldsen et al. (2013) introduce a sort of matrix where impacts are first categorized in the operational, tactical and strategic level (as described in project management literature), and then the impacts are further categorized in alignment with the three-pillar thinking (as used in the SIA), where also the unintended impacts are included.

3. Research methods

Not surprisingly, the initial literature review undertaken in the work leading up to this paper revealed an extensive literature to different aspects of the workings. As commented by Hacking and Guthrie (2008, p. 74), ‘[a] difficulty when considering [indicator assessment] is not the scarcity of literature, but rather the vast quantity’. The approach taken was to continue the review until what Corbin and Strauss (2008, p. 143) call ‘theoretical saturation’ – that is, the point at which new properties, dimensions or relationships do not seem to emerge – was achieved.

As Bond et al. (2011) comment, sustainability is a value-based concept. What sustainability is and by which indicators it ought to be measured vary among stakeholders and the public. Philosophers have long been aware of this challenge, and formulated it according to different conceptual frameworks. Gadamer (1960), for instance, describes this as the horizon of understanding particular to each individual, while Taylor (2004) describes how these particularities of individuals form so-called social imaginaries that precede and determine how different groups perceive phenomena in separate manners. Bond et al. (2011) underline how a pluralistic approach proves necessary to remediate this to the extent possible. The research reported in this paper took such a pluralistic approach, assembling the students in groups of approximately five in order to allow for differentiated perspectives while holding the material manageable. Bond et al. (2011) equally stress the need for the assessment process to be carried out throughout the project, so that stakeholders are not marginalized in the implementation phases. As our research concerned an exercise with a limited time frame, such follow-up proved not possible.

In order to gather empirical support for the answers to our two research questions, we asked 538 of our engineering students – assembled in 121 groups – to suggest indicators that can be used to assess a major road project. We have also analysed the propensities of an exemplar SIA performed by professional project analysts. Then we sorted the indicators actually used into categories. This sorting was done in a subjective way by the authors. This subjective categorization is certainly vulnerable for misinterpretation, but since we only want to show obvious trends, we consider our subjective categorizations to be valid enough.

The decision-support framework procedures of impact assessments can, for instance following Donnelly et al. (2006, pp. 152–153), be divided into four logically separated steps. First, the objectives and targets need to be clarified; second, choose what indicators are to be assessed; third, identify available data; and, fourth, consider if the data are in suitable form or if a new monitoring system should be established (or if suitable proxy data or alternatives should be identified). The students constituting our group of respondents carried out their analysis according to all of these four steps. The results presented in the current paper consider solely the second of these steps, as they reveal what we consider to be a weakness of the SIA as described by the OECD (2010).

3.1. The student's SIA, based on the guidance of OECD (2010)

A total of 538 undergraduate students, assembled into 121 groups, were asked during the autumn semesters in 2010, 2011 and 2012 to identify indicators that could be used to assess a major infrastructure project in Norway called E6 Øst. For this, they were not given any list of wanted indicators to pick from, but they were asked to suggest the indicators relevant for an assessment of this project, within a short and long time frame. Prior to the task, they received a four-hour lecture on how to carry out an SIA, and then a four-hour lecture about formulating purpose (on strategic level), objectives (on tactical level) and project results (on operational level). They also received access to project documents, information on the project website, a guest lecture presenting the project and an excursion to the construction site. We realize that when 121 groups are asked to invent indicators, when they in general experience a high workload, there is a chance that they put different efforts into it. This is probably the reason why some groups identified many indicators, while others had few. Overall, we presumed the students capable of understanding the complexity of major investment projects in general, and to understand what impacts may be considered relevant in the case of this specific road project.

The assignment was compulsory, and they received a score that impacted their final course grade. Although the students were from the same study programme and the same age group, we find them to be representative enough to the majority of the Norwegian population to permit for analysis. The Norwegian University of Science and Technology (NTNU) – the university among whose students the research took place – is the only polytechnic university in Norway, recruiting from all socio-economic backgrounds (due to generous financing of study), regions, gender (40% female) and religions. The students were fresh at university, and thereby not yet exposed to the embodied institutional thought patterns.

3.2. The professional SIA, the exemplar assessment of OECD (2010)

We have also scrutinized an SIA assessment of a major investment project performed by experts in the field. According to the Canton of Bern (2008, p. 2), this SIA intends to provide an ‘overall balance of the impacts of a planned project on local or regional sustainable development’.

Several SIAs carried out by professionals exist. There are three arguments for scrutinizing just this SIA from the Canton of Bern. The first is that OECD (2010) presents it as an exemplar SIA. The second is that the SIAs carried out before the latest version of the guidance (OECD 2010) was published fall outside our project of analysing SIAs in the light of this. The third is that the other SIAs we have gained access to, from the years following 2010, do not explicitly acknowledge being conducted exclusively according to this guidance. Therefore, an analysis of other SIAs would not have the level of precision necessary to articulate the shortcomings and corresponding possibilities for improvement of the guidance from OECD (2010).

4. Findings

Our findings primarily concern our first research question about what indicators are actually chosen as the most significant in SIAs. As mentioned, we have let 538 of our undergraduate students, assembled in 121 groups, identify indicators for an SIA on the major public road investment project E6 Øst in Trondheim, Norway. After that, we examined an SIA of a project initiated to link two major ski areas in Switzerland. This SIA is presented as an exemplar one by OECD (2010).

4.1. E6 Øst in Trondheim, Norway

The 121 respondent groups identified a total of 984 indicators pertinent to the investment project in subject. One group identified four indicators, while the most ambitious groups identified 18. We assembled the identified indicators in categories based on what was subjectively judged strategic, tactical and operational concerns, with sub-divisions within the categories according to the three-pillar thinking of economic, social and environmental indicators. The frequencies of the 984 indicators, including duplicates, within these nine categories were systemized in Figure 2.

Figure 2 Frequencies of indicators in the E6 Øst project, without duplicates (138 indicators used totally 984 times, sorted in 9 categories).

In order to allow comparison with the frequencies of indicators in the project linking two ski areas in Bern, we removed the duplicates. After that, we were left with 138 different indicators. The frequencies of the 138 indicators were systemized in Figure 3.

Figure 3 Frequencies of indicators in the E6 Øst project (138 indicators used totally 984 times, sorted in nine categories).

4.2. Linking two ski areas in Bern, Switzerland

As we can observe in Figure 4, a total of 124 indicators were chosen to assess the most important impacts of the project. Again, the indicators were assembled in categories based on what was judged strategic, tactical and operational concerns, with sub-divisions within the categories according to the three-pillar thinking of economic, social and environmental indicators. The frequencies of indicators within these nine categories were systemized in Figure 4.

Figure 4 Frequencies of indicators in the project linking two ski areas in Bern (124 indicators sorted in nine categories).

On the surface, the SIA presented a relatively clear picture. The project scored high on the economic indicators, relatively neutral for the social indicators and very low on the environmental indicators. The overall assessment was slightly negative. Due to this negative score, the analyses recommended that the project was not carried out.

When looking at the original categorization into only the three categories of economic, social and environmental indicators, we found indicators for what must be considered very local impacts – quality of materials – alongside indicators for impacts that must be considered in a far broader perspective, even global – the impacts on biodiversity and climate. We can follow this tendency to mix local and more general indicators throughout the other pillars of sustainability. For instance, cost internalization and promotion of economic development are juxtaposed in the economic section, while quality of the built-up areas are juxtaposed in the social pillar.

5. Discussion

Our discussion combines the findings from the literature study with the findings of what indicators are actually chosen as the most significant. From this, we will answer the second research question about how the assessors should choose their indicators.

It seems to be commonly accepted that sustainability has three main dimensions, namely the economic, social and environmental dimension. It is incorporated in the SIA that all impacts can be sorted in one of these three dimensions.

Our experience from the student assignments is that it is not controversial to categorize indicators into the three dimensions. When we take it one step further, and sort the students’ indicators according to whether they take the strategic, tactical or operational perspective, we see a source of controversy. The students have completely neglected to include operational environmental impacts when suggesting indicators. They have relatively few indicators on operational social impacts, and they have paid little attention to strategic, economic impacts. When looking at the proposed indicators in retrospect, it is more or less obvious that when the national public road authority initiates a road investment project, the strategic, economic impacts are important. When carrying out an SIA on a public road investment project, the assessor must consider the economic impacts from the initiating party's perspective.

The first observation from the exemplar SIA is that it makes sense to categorize the indicators into the economic, social and environmental dimensions. When sorting the economic, social and environmental indicators further into the strategic, tactical and operational categories, we find no operational social indicators. Only one of the 124 indicators – the indicator ‘Improvement in product life’ – deals with operational, economic impacts. There are no indicators in the exemplar SIA that looks at investment costs. When assessing sustainability, it is important to look at the investment project in a broad manner. Though, regardless of who the initiating party is, the investment cost will influence the sustainability. Just as in the answers to the student assignments, the professional SIA has omitted important indicators.

Our findings from project management literature indicates that two fundamental questions should be addressed before initiating an SIA, notably (1) from what perspective sustainability is to be assessed from and (2) over what time span is one to assess the sustainability. Both questions may seem easy to deal with, but they do in fact involve considerable challenges.

Addressing from what perspective sustainability is to be assessed might seem straightforward at first, corresponding, for instance, to the perspective of the contractor, end user or initiating party. However, all these perspectives are interlinked, so that what may be considered sustainable from one perspective does not necessarily prove sustainable from another perspective. For instance, driving an electric car can be judged very sustainable in the end user perspective, since it will reduce local CO₂ emissions (environmental pillar), increase mobility (social pillar) and provide cost-efficient transportation (economic pillar). However, from an international perspective the problems linked to battery production, car availability, electric power production and other factors may far outweigh such local advantages. The recommendations from an SIA can be influenced by what perspective the assessor has.

If the assessors that perform an SIA consider that the sustainability of an investment project can be assessed by different stakeholders, it may be easier to control that the chosen indicators actually cover all relevant perspectives. If a major road investment project is considered as a sustainable project locally, this will not help much if it is not sustainable in a national perspective.

Addressing the time frame over which sustainability occurs is equally a factor to the assessment. In the same way as we propose that the assessors that perform an SIA should consider from what perspective sustainability is to be assessed, we also propose that they specify which time frame their SIA covers.

That we acknowledge that the results of an SIA necessarily will be marked by the assessors' subjectivity does not imply that we acknowledge that analyses based on (mostly) qualitative procedures are too biased in nature. The aggregation and following presentation of subjectively assessed impacts of an investment project can enlighten the decision-making process.

Both the questions about which perspective an indicator refers to, and about what time frame it refers to, may be answered by guiding the assessors to sort the impacts on strategic, tactical and operational levels as suggested in project management literature. The strategic level refers to the – normally long-term – regional priorities and policies. The tactical level refers to the local community together with its present and future needs. The operational level refers roughly to the construction site and period. These questions come in addition to the question about which of the economic, social and environmental dimensions an indicator should be categorized in when presenting the result of an SIA.

6. Conclusions and recommendations

Policies (strategic) constrain plans (tactical) which constrain projects (operational), and separate SIAs are typically conducted at these different levels. When selecting indicators for an SIA of a project, the assessor must have in mind that the project is initiated in order to support an overall policy. Our findings on SIA assessments – from our students' assessment of the E6 Øst project and from when professionals assessed the project linking two major ski areas in Bern – gave the impression that as long as no extra precautions are taken, the selection of indicators is left to the discretion of the assessors. That may lead the assessors to select the most tangible and familiar indicators, while they may not take notice of important impacts that are imprecisely described or difficult to foresee. The assessment can become more subjective, something that will reduce the predictability and thereby the quality of the outcome. We found little or no systematic reflection on what indicators to choose neither in the E6 Øst project nor in the project linking two major ski areas in Bern.

We also found that indicators relating to different perspectives, be it the perspective of the contractor, the end user or the initiating party, were juxtaposed without discrimination. Operational concerns (noise during construction period) and strategic concerns (greenhouse gas emissions) were found mixed together in a seemingly random manner. Noise during the construction period can cause a temporary annoyance for the nearby neighbours, but most probably they can live with it. Greenhouse gas emissions can cause permanent global damage. In short, we consider neither of the studied sets of SIA indicators to be perfect.

An SIA is – as we have seen – intended to cover quite a broad spectrum of impacts. Many of the objections to assessments of sustainability in the literature concern how difficult it is to weight, measure and price impacts. We find it necessary to address a challenge that seems less widely discussed in the literature. The choice of indicators to include will in most cases impact on the outcome of the analysis, so an assessor of sustainability must identify all relevant indicators before getting concerned with the pricing. Our ambition has been to establish a certain theoretical clarity concerning what one is to assess when performing an SIA.

Some sort of pre-qualification of which type of indicators to choose ought to exist. An intuitive approach to the establishment of criteria runs the risk of omitting potentially fundamental sectors of analysis. We suggest a set of nine different categories of SIA indicators that can be used as a framework for such a pre-qualification, which is illustrated in Figure 5.

Figure 5 Nine different categories of SIA indicators.

If the assessors have a firm framework they can relate to when choosing indicators, the assessments can be more objective. When we suggest this framework, we are still aware of the principle of ‘proportional analysis’. The depth and scope of the impact assessment should match the significance of the investment project. An analysis of different possible gravel qualities is not of prime interest when assessing the sustainability impacts of an infrastructure investment project.

To conclude, this paper has two main contributions. First, we point out how easy it is to unconsciously omit important indicators in an SIA. If it happens, it will significantly reduce the quality of the SIA. Second, we have proposed a framework with nine different categories that will help assessors to identify a more exhaustive list of impact indicators than if they only relate to the economic, social and environmental dimensions.

Notes

^1. SIA (sustainability impact assessment, based on three pillar thinking) should not be confused with social impact assessment – also abbreviated SIA – which reviews only the social effects. We consider SIA (sustainability impact assessment), as described by OECD (2010), to be one of several possible sustainability assessment tools.

^2. For a discussion on the visual representations of sustainability, see Adams (2006).

^3. A wide array of definitions exists. In fact, their multiplicity illustrates not only how complex the concept of sustainability is to define, but also the widespread experienced need for a definition. Hasna (2010) enlist 67 definitions from the plethora (strangely omitting the OECD definition, which constitutes the conceptual framework of our work).

^4. This paper is inspired by a three-year research project, examining how and to what extent sustainability is taken care of within concept appraisal for public investments in a quality assurance scheme governed by the Norwegian Ministry of Finance. Today, public investment projects with estimated cost surpassing NOK 750 million (approximately $130 million) are subjected to scrutiny. We refer to projects of this magnitude when we in the following use the term investment project.