Developing an integrated sustainability and resilience framework of indicators for the assessment of low-carbon energy technologies at the local level

ABSTRACT

Sustainability indicators have been broadly used to assess energy technologies both at the national and local levels. However, very few studies have addressed the issue of resilience of energy technologies. Moreover, there is a lack of an integrated framework that combines both sustainability and resilience indicators for the assessment of energy technologies. The aim of this paper is to present the development of an integrated framework of sustainability and resilience indicators for the assessment of low-carbon energy technologies at the local level in Europe. The selection of indicators is based on a modified ‘3S’ approach, composed of literature review, self-validation, scientific validation, and social validation. The study incorporates local stakeholders’ feedback on the selection and validation of evaluation criteria based on a European survey. The vast majority of respondents approved and validated the indicators that were selected through the internal and experts’ validation steps.

KEYWORDS:

• Sustainability and resilience indicators
• indicators selection and validation
• integrated assessment framework
• low-carbon energy technologies
• local stakeholders

1. Introduction

Important policy and investment decisions should be made regarding current and future low-carbon energy technologies that would be deployed in the coming years and decades. The inclusion of multiple sustainability criteria in the assessment process is of major importance and a growing priority of the EU. Recent studies have considered environmental externalities (Roth 2009; Weinzettel, Havránek, and Scasny 2012) as well as social, economic and technological aspects of energy systems' sustainability (Chatzimouratidis and Pilavachi 2008; Schenler et al. 2009; Evans, Strezov, and Evans 2009; Carrera and Mack 2010). Moreover, there is an increasing load of literature on evaluating the European energy security of supply (Chevalier 2006; Constantini et al. 2007) and its vulnerability on fuel imports (Gupta 2008; Bhattacharyya 2009; Roupas, Flamos, and Psarras 2009, 2011).

Issues such as disaster risk management of energy systems (McLellan et al. 2012), the increasing reliance on fuel imports and risks involved for the EU security of energy supply (European Commission 2014) and the likely impacts of a changing climate to the European energy system (Dowling 2013) constitute resilience of current and future energy technologies as a major priority for the EU. The concept of system resilience has been contextualised in different fields (Holling 1973; Tyler and Moench 2012; Collier et al. 2013). In the energy sector though there are few studies that explicitly address system resilience (Gaudreau and Gibson 2010; O'Brien and Hope 2010; McLellan et al. 2012; Molyneaux et al. 2012).

Moreover, various literature acknowledge the desirability of integrating sustainability and complex systems (Fiksel 2006; McLellan et al. 2012). While there are already attempts to integrate these two components (Milman and Short 2008; O'Brien and Hope 2010; Molyneaux et al. 2012), there is a lack of literature on sustainability and resilience indicators integrated in one framework for the assessment of energy technologies.

This paper presents the development of an assessment framework of indicators for low-carbon energy technologies that integrates both sustainability and resilience aspects. More specifically, the focus of this paper is on the selection of criteria and indicators (envisioning stage), going through the steps of self-validation (internal), scientific validation (by experts), and social validation (by stakeholders). The other phases of the overall assessment methodology and application such as impact assessment and modelling, stakeholders' weights elicitation and final evaluation are presented and discussed in other articles (Grafakos et al. 2010; Grafakos, Enseñado, and Flamos 2015; Grafakos, Enseñado, Flamos, and Rotmans 2015).

The paper is structured as follows: Section 2 discusses the main concepts, approaches, and aspects when developing an integrated sustainability assessment (ISA) framework of energy technologies. Section 3 describes the main methodological steps and data collection methods of the current study on stakeholders' validation of criteria. Section 4 presents the results of the refinement and validation process of criteria and indicators. The final section discusses the main implications of the research findings, future research directions, and concluding remarks.

1.1. Integrated sustainability assessment

• Numerous approaches and frameworks have been developed to measure and assess the achievement of sustainable development goals, ranging from guidelines to more specific indicator-based frameworks (Ness et al. 2007). For a comprehensive review and elaborated classification of different sustainability assessment methodologies, see Ness et al. (2007) and Singh et al. (2012). According to Ness et al. (2007), many of the integrated assessment tools that have been reviewed integrate environmental and social aspects of sustainability. According to Weaver and Rotmans (2006) the dimensions of integration that can take place in sustainability assessment processes are: integrated objectives that embrace multiple sustainability concerns and values

• knowledge and information across multiple domains and sources

• sustainability values and principles throughout the process

• stakeholders, policy-makers and experts

• quantitative and qualitative tools, methods, information

• proposal design and assessment (integration into policy development process)

• social learning, self – evaluation and reflexivity

• internally ISAs to form a coherent assessment regime

The main phases of an ISA are scoping, envisioning, experimenting and learning, and monitoring and evaluation (Weaver and Rotmans 2006). Multiple criteria analysis (MCA) approaches have been classified under the integrated assessment category since they integrate multiple objectives while including multiple stakeholders in the assessment process. Along these lines, we developed an overall MCA assessment framework of energy technologies that is aligned with the main phases of the ISA.

The main components and stages of the MCA assessment framework are illustrated at Figure 1. The focus of this paper is on the selection of criteria and indicators (envisioning) stage of the overall assessment process which is illustrated in more detail in Figure 2.

Figure 1. Integrated MCA sustainability assessment framework for energy technologies.

Figure 2. The selection and validation process of indicators.

1.2. Selecting indicators for assessment

The use of indicators to measure progress and track trends towards specific policy objectives (Cobb and Rixford 1998) has been extended widely to numerous sustainability assessment frameworks in the last two decades. Indicators can be used not only to determine baseline conditions (state) and current performance, predict future trends, but also to function as monitoring and warning systems. Indicators can also be used for making comparisons (across time and space or with targets), in performance review, and for improving scientific and policy understandings (Gallopin 1996; Cool Stephen and George Stankey 2004; Hezri and Dovers 2006).

The selection and validation of evaluation criteria and indicators is an important part of any environmental assessment and decision-making process, including energy and climate change mitigation evaluations (Bockstaller and Girardin 2003; Cloquell-Ballester et al. 2006). Based on the literature that delves on measuring energy sustainability, it has been emphasised that there is no particular indicator framework that is suitable to all applications (Keirstead 2007). Hence, it is necessary to take into account the intended goals for the use of the criteria. Moreover, criteria have to be chosen selectively in order to maximise their effectiveness and relevance (Keirstead 2007).

However, when researchers and analysts apply a multiple criteria or multiple indicator assessment framework, they often neglect this very essential stage of the decision-making process. Criteria and indicators are usually applied intuitively by the analysts (Hak, Kovanda, and Weinzettel 2012). According to Bockstaller and Girardin (2003), many indicator developers do not consider the validation of indicators, probably because they assume that long-term acceptance of indicators by users suffice to indicate their credibility. Experts often attempt to deduce stakeholders' preferences instead of including them directly in the decision-making process. According to Kowalski et al. (2009), most applications on energy issues focus on technical aspects. Also, these generally do not involve stakeholders in the decision-making process in a systematic and participatory way.

Cloquell-Ballester et al. (2006) developed the ‘3S’ methodology for validating indicators in the field of environmental studies. The current study modifies the ‘3S’ validation methodology in the context of low-carbon energy planning for the selection and refinement of sustainability indicators. Including stakeholders at the initial stage of the decision-making (e.g. selection of evaluation criteria) is imperative for a participatory process. With issues on public acceptance, stakeholder participation is crucial to guarantee success as well as stability of energy supply systems (Braune, Pinkwart, and Reeg 2009).

1.3. Sustainability of energy systems

The International Atomic Energy Agency was the first to attempt assessing the energy sector from a sustainability angle.

The original framework considered the economic, social, environmental, and institutional dimensions of sustainable development.

This indicator-based framework focused on the assessment of the overall energy system at the national level for comparative purposes between countries and did not aim to compare different energy technologies. Many authors since then have attempted to build sustainability assessment indicator frameworks for evaluating energy technologies against the three pillars of sustainability: environmental, economic, and social.

However, even a universal assessment framework of indicators of energy technologies that has been attempted already (Chatzimouratidis and Pilavachi 2008; Evans, Strezov, and Evans 2009) would not be applicable in all cases and geographical contexts.

Afgan, Carvalho, and Hovanov (2000) developed a multiple criteria assessment framework of indicators for energy technologies including environmental, resource, social, and economic indicators. Afgan, Carvalho, and Hovanov (2000) extended the criteria categories by adding the resource category in order to capture the inflows (resources, fuels) and out flows (emissions) of energy technologies. Evans, Strezov, and Evans (2009) conducted an assessment of energy systems by developing a sustainability framework of selected criteria relevant to the Australian context.

While most of the indicator frameworks have been developed to assess the sustainability of energy system at the national level, few studies have been conducted to establish sets of indicators at the local level. Del Rio and Burguillo (2008) developed an integrated framework of indicators for the assessment of the impact of renewable energy deployment on local sustainability, whereas Kowalski et al. (2009) developed a comprehensive list of indicators to assess different sustainable energy scenarios at the local level.

Donkelaar ten and Amara (2010) came up with the 20 most used assessment criteria for energy projects (Appendix 1) based on an extensive literature review. Most of the criteria identified are classified under the environmental category, showing the importance of environmental/ natural resources issues of energy technologies.

While the aforementioned approaches attempted to address all sustainability aspects of energy technologies, social aspects were taken into account to a lesser extent than economic and environmental ones. Assefa and Frostell (2007) and Carrera and Mack (2010) emphasised social sustainability issues more prominently, bringing multiple social issues in the assessment process of energy technologies.

Shen, Li, and Yuan (2010) gauged three goals, namely energy, environmental, and economic. The study adopted the framework proposed by Komor and Bazilian (2005) in preparing a model for assessing renewable energy sources. Moreover, other research studies were used as reference. These studies emphasise the energy and technology system aspects in the assessment process.

Technology defines, to a large extent, the demand for material and energy flows and needs for infrastructure while converting mass flows of materials to emissions and wastes. Technology could drive innovation and influence production and consumption patterns, lifestyles, and social relationships while minimising environmental impacts and risks (Musango and Brent 2011). Therefore, the development and use of technological systems is highly interrelated with the other aspects of sustainability. Technology is embedded strongly in the economic, social subsystems within the broader environmental system that societies and economies operate. Technologies and sustainable development sub-systems mutually influence each other (Gaziulusoy, Boyle, and McDowall 2008) and Musango and Brent (2011).

Various authors attempted to include technological aspects of energy systems in the sustainability assessment frameworks either from an energy system perspective (Lund 2009; Shen, Li, and Yuan 2010) or from a technology market perspective (Lee, Yoon, and Kim 2007; Lewis and Wiser 2007), expanding and improving the existing sustainability assessment frameworks. These criteria related to energy system were an attempt to incorporate implicitly system resilience aspects in the sustainability assessment frameworks of energy systems.

1.4. Resilience of energy systems

According to Fiksel (2006), the complexity, dynamics, and nonlinear nature of the interdependent system that the static view of ‘sustainability’ as a steady-state equilibrium does not reflect the reality. Technological, geopolitical, or climatic changes will inevitably disrupt the cycle of material and energy flows. Achieving sustainability from a broader and long-term perspective, Fiksel (2006) argues, will require the development of resilient, adaptive industrial, and societal systems.

In a similar manner, Milman and Short (2008) argue that indicators measuring urban sustainability have a narrow focus and solely describe the current state of the urban system.

Following the systems thinking approach that Fiksel (2006) and Milman and Short (2008) suggested, O'Brien and Hope (2010) attempted for the first time to conceptualise a resilient energy system and explored how to incorporate resilience considerations into the energy system.

Molyneaux et al. (2012) developed specific resilience indicators and a composite Resilience Index for estimating and comparing the resilience of the electricity systems of different countries. According to Molyneaux et al. (2012), a robust power system is an essential component of a country's functioning economic system.

Looking at the numerous studies on sustainability assessment of energy technologies, along with the growing literature on resilience and systems thinking approach to energy systems, this study aims to combine sustainability and resilience aspects of energy technologies in an integrated Multiple Criteria Assessment framework for improving decision and policy-making support at the local level in Europe. Participatory MCA approaches in energy planning and assessment can facilitate a decision-making process that could lead to social and stakeholders' learning (Stagl 2006), an important element of resilience and adaptive management as has been emphasised by O'Brien and Hope (2010).

According to the aforementioned studies we identify the following key elements of resilience of energy technologies to be incorporated in our proposed framework of criteria and indicators: redundancy, efficiency, diversity, risks reduction, and adaptive capacity.

1.5. Inclusion of stakeholders in the selection of evaluation criteria and indicators

At the European level, the New Energy Externalities Development for Sustainability (NEEDS) project applied a Multiple Criteria Decision Analysis of energy technologies in four European countries, namely France, Germany, Italy, and Switzerland for the year 2050 (Hirschberg et al. 2007; Makowski et al. 2009).

At a national level, MCA was applied to evaluate future energy policy options in the UK (Stagl 2006).

At a local level, in Urnasch, a municipality in Switzerland, a stakeholder-based MCA was carried out to assess future energy systems.

 Table 1 summarises the multiple criteria studies that have been applied at the local level to assess different energy technologies and options.

Table 1. Local case studies of sustainability assessment of energy technologies/options.

Local context	Study	Thematic area	Current vs. future energy options	Criteria selection	Real application
Chios Island, Greece	Haralambopoulos and Polatidis (2003)	Renewable energy projects	Current	By researchers	Proposed methodology
Sardinia Island, Italy	Beccali, Cellura, and Mistretta (2003)	Renewable energy technologies	Current	By researchers	Proposed methodology
Salina Island, Italy	Cavallaro and Ciraolo (2005)	Wind energy plants	Current	By researchers	Yes
Catalonia, Spain	Gamboa and Munda (2007)	Wind farm locations	Current	By researchers and stakeholders	Yes
Metropolitan Borough of Kirklees, UK	Burton and Hubacek (2007)	Small-scale energy technology applications	Current	By researchers	Yes
Local case study, Norway	Loken, Botterud, and Hoken (2009)	Future energy-supply infrastructure	Future	By researchers	Proposed methodology
Crete, Greece	Tsoutsos et al. (2009)	Sustainable energy planning	Current	By researchers	Yes
Urnasch, Switzerland	Trutnevyte, Stauffacher, and Scholz (2011)	Future energy systems	Future (2035)	By researchers and stakeholders	Yes
Thassos, Greece	Mourmouris and Potolias (2013)	Renewable energy sources	Current	By researchers	Proposed methodology
Crete, Greece	Kaldellis, Anestis, and Koronaki (2013)	Strategic electricity generation planning	Current	By researchers	Yes

In most of the cases at the local/regional level as it is illustrated in Table 2, the inclusion of stakeholders in the selection or validation of evaluation criteria and indicators was not manifested. The selection of criteria and indicators for assessing current and future energy technology options was undertaken exclusively by the researchers. Furthermore, it is clear that there is a need to incorporate local stakeholders' perspectives while developing an overall framework of indicators for assessing present and future energy technologies. The inclusion of stakeholders' varied interests in the planning and decision-making process facilitates long-term commitment and cooperation in implementing energy alternatives (Tsoutsos et al. 2009). Moreover, involving different stakeholders in the energy planning and decision-making process already from the initial stage of selecting indicators or assessment increases legitimacy, facilitates learning, and allows for the inclusion of multiple perspectives (Braune, Pinkwart, and Reeg 2009; Van der Gaast, Begg, and Flamos 2009; Gamper and Turcanu 2007). The current study includes local stakeholders in the process of indicator selection and validation assuring that their perspectives would be reflected in the integrated assessment framework.

Table 2. Final validated set of criteria and indicators.

Criteria categories	Indicators	Description
Environmental	CO2eq emissions	The indicator reflects the potential impacts of global climate change caused by greenhouse gas (GHG) emissions for the production of 1 kWh
	Climate resilience	The degree of resilience of the energy technology to the future climatic changes and extreme weather events
	Noise pollution	Part of population feeling highly affected by the noise caused due to the function of the energy facility. This indicator is case sensitive and could have been measured as a factor of the noise generation by the energy technology estimated in dB multiplied by the number of people affected by the noise. However since we are investigating different energy technologies and systems at a European scale we cannot measure precisely this indicator and therefore we will use an ordinal relevant scale to measure the perceived noise
	(Radioactive) waste	Amount of (radioactive) waste generated by the plant divided by energy produced
	Waste disposal (infrastructure)	Waste generation during the life cycle of the fuel and technology or availability of waste disposal infrastructure
	Ecosystem damages	This criterion quantifies the impacts to flora and fauna due to acidification and eutrophication caused by pollution from the production of 1 kWh electricity by the energy system and technology
	Land use requirement	The land required by each power plant and technology to be installed
	Fuel use	Amount of fuel use per kWh of final electricity consumption
Social	Level of public resistance/opposition	Energy system induced conflicts that may endanger the cohesion of society (e.g. nuclear, wind, CCS). Opposition might occur due to the perceptions of people regarding the catastrophic potential or other environmental impacts (aesthetic, odour, noise) of the energy technology/system. This indicator also integrates the aspect of participatory requirement for the application of the technology. The higher the public opposition, the higher the participatory requirement is
	Aesthetic/functional impact	Part of population that perceives a functional or aesthetic impairment of the landscape area caused by the energy system. The aesthetic impairment is judged subjectively and therefore this criterion fits in the social category than the environmental one. In addition this is also a very location specific indicator and therefore an average metric will be determined measured in relative ordinal scale
	Mortality and morbidity	Mortality and Morbidity due to air pollution caused by normal operation of the technology. This indicator is considered as an impact and composite indicator since it integrates all human health impacts caused from air pollutant emissions as NOx, SO2 and PM
	Accident fatalities	Lost of lifes of workers and public during installation and operation. Surrogate for risk aversion. This criterion partly integrates the catastrophic potential of the energy system/technology
Economic	Levelised costs (incl. capital, operational and maintenance, fuel costs)	Levelised costs of energy (LCOE): investment costs, operational and maintenance costs, capacity factor, efficiency, material use
	(Local) employment	The extent to which the application of the technology can create jobs at the investment, operation and maintenance stage. Furtheremore the criterion of employment reflects partly the extent of the impact that the technology has to the local economic development by providing jobs and generating income
Energy	Energy cost stability/sensitivity to fuel price fluctuation	The sensitivity of technology costs of electricity generation to energy and fuels prices fluctuations. The fraction of fuel cost to the overall electricity generation cost
	Stability of energy generation	Stability of output of electric power generated depending on the technology used. This reflects whether the energy supply is being interrupted. The presence of these interruptions impacts the electricity network stability. This criterion reflects whether the energy supply faces any interruptions due to the type of energy technology
	Peak load response	Technology specific ability to respond swiftly to large variation of demand in time / % representing the possibility to satisfy the required load
	Market concentration on supply	The market concentration on the supply of primary sources of energy that could lead to disruption due to economic or political reasons
Technological/Market	Technological maturity	The extent to which the technology is technically mature. The criterion refers to the level of technology's technological development and furthermore the spread of the technology at the market
	Market size (domestic)	Demand for final products (of energy technologies) and potential market size domestically. The potential market size plays an important role to establish industrial competitiveness and stimulate economic growth
	Market size (potential export)	Demand for final products (of energy technologies) and potential market size internationally
	Innovative ability	Flexibility and potential of the technology to integrate technological innovations

2. Criteria selection and validation process

In the current study, an extensive literature review on evaluation criteria and indicators that have been utilised in previous studies was conducted. The commonly used criteria and indicators were adopted, and a few more were added in the selection. The combination ultimately resulted in a new criteria and indicators framework for the evaluation of future low-carbon energy technologies in Europe. In that sense, the set of criteria and indicators needed validation from and refinement by the actual users and stakeholders.

The study modified the ‘3S’ indicators’ validation methodology developed by Cloquell-Ballester et al. (2006) and applied it to the current research context by undertaking the following five steps for selecting and validating indicators:

• Extensive literature review

• Screening of indicators

• Self-validation and refinement (based on rigorous internal peer review),

• Scientific validation and refinement (based on experts review), and

• Social validation and refinement (based on a survey of local stakeholders).

Figure 2 illustrates diagrammatically the five steps of the selection and validation of indicators (envisioning) stage of the overall integrated MCA sustainability assessment framework that was depicted in Figure 2. As Cloquell-Ballester et al. (2006) argue, these validation stages are complementary so that the indicators' credibility and usability increases as we complete and move from one validation stage to the next. Furthermore, the proposed framework suggests an application of a participatory Multiple Criteria Assessment framework for energy technologies aiming at the active participation of different stakeholders during its actual application, that would lead to an adaptive decision-making process.

2.1. Extensive literature review

The initial selection of evaluation criteria and indicators was based on an extensive literature review of studies in the field of low-carbon energy planning and ISA of energy options. The literature review included both indicator-based approaches of sustainability assessment of energy technologies and non-indicator-based frameworks addressing sustainability and resilience aspects of energy systems.

2.2. Screening of indicators

During the selection process, the evaluation criteria and indicators were screened. In particular, each indicator was filtered through specific attributes as those have been described by Belton and Stewart (2002), Keeney and Gregory (2005), and Grafakos et al. (2010):

• Operational: Being able to specify how well each mitigation option meets the objectives expressed by the evaluation criteria.

• Value relevant: Linking the concept of each criterion to the final objectives it is meant to represent. In other terms, it presupposes that an objective is comprehensively described by underlying criteria.

• Decomposed: Possibility to break down an objective into specific means.

• Reliable: A malfunctioning criterion should not render the whole set of criteria unworkable.

• Measurable: Degree of measurement of the performance of alternatives against specified criteria.

• Non-redundant: Limiting the number of criteria addressing the same objective, meaning avoidance of duplication of information in criteria.

• Minimum in size: The number of criteria employed should be only the absolutely necessary to provide representation of policy objectives.

• Complete: The set of criteria should be complete in order to capture all the key aspects of the objectives.

• Understandable: The selected criteria should be simple to comprehend not only by experts in the relevant field but by non-specialists as well.

• Preferential independent: Preferences associated with the performances of each option should be independent of each other from one criterion to the next.

• Comprehensive: The criteria in the selection should cover and/or relate to the different objectives, and that implicit value judgements are suitable to the decision problem.

• Direct: The set of criteria selected should directly be linked to the objectives, and that there are no controversial implications between tradeoffs.

• Unambiguous: Each of the criterion should be precise in its definition (i.e. how it describes or measures the elements involved).

In addition to these general conditions, we introduced a few more attributes that specifically apply to ISA of low-carbon energy technologies in Europe at the local level:

• Geographical coverage and local context: The criteria should be applicable in Europe at the local level.

• Data availability: There should be available data or, in its absence, data collection methods.

2.3. Self-validation and refinement

After an extensive literature review and screening of the initial long list of indicators against the aforementioned attributes, the authors initiated the internal validation process. Researchers working on the study reviewed 40 preliminary indicators that passed the first screening. Afterwards, the researchers conducted several meetings that led to the refinement and selection of a set of 33 indicators. These indicators were classified under five criteria categories: environmental, social, economic, energy, and technology (market). Both sustainability and resilience criteria were embedded in the criteria categories.

2.4. Scientific validation and refinement

The set of 33 indicators were then reviewed by 10 European experts in the field of energy planning and energy technology assessment for further refinement and feedback. The experts participated in this process through email and phone communication and by completing a questionnaire. The experts expressed their views on whether they agree or disagree with the selection of indicators and if they could suggest any adjustments.

Furthermore, experts had the option to suggest additional indicators to be included in the preliminary set. The experts could also add their comments and recommendations for further improvement of the indicators. The research team asked for clarifications in cases where it was deemed necessary, incorporated experts' comments and feedback, and made adjustments to the set of indicators accordingly. After the experts' validation and further internal discussions, the set of indicators came down to 22 classified in five different categories including both sustainability and resilience-related indicators.

3. Categories of criteria and indicators

3.1. Environmental category

Shen, Li, and Yuan (2010) highlighted the significance of carbon emissions reduction, environmental sustainability, SO_x and NO_x emissions reductions, and low-land requirements. It has been established through numerous studies that CO₂ emissions of energy system is an important criterion in assessing energy technologies. In addition to the impact-related aspect of CO₂ emissions regarding their contribution to climate change, this criterion entails also a risk aspect regarding the potential of further expansion of carbon pricing. It is therefore important to be able to account for the vulnerability of energy technologies to increases of the energy prices due to the potential of carbon pricing (Molyneaux et al. 2012).

Reduction of local air pollutants, such as SO_x and NO_x emissions, has been recommended by many studies as an evaluation criterion of energy technologies (Diakoulaki and Karangelis 2007). Environmental sustainability, within the context of electricity, refers to the shift from fossil fuels to, justifiably, renewable energy. However, the evaluation of the impacts brought by the use of renewable energy should be according to noise pollution, landscape impact, microclimatic changes, and unpleasant odours (Beccali, Cellura, and Mistretta 2003 in Shen, Li, and Yuan (2010). In SF Energy Invest (2010), specific criteria were included under the environmental dimension. These include waste creation and disposal, including hazardous waste, noise, and land use. Low-land requirement has also been cited by different studies (e.g. Afgan, Carhalho, and Hovanov 2000; Beccali, Cellura, and Mistretta 2003; Andrews et al. 2011) as an important criterion. This is due to the fact that demand for land can cause economic losses which are comparative to the site value (Shen, Li, and Yuan 2010). The issue of climate resilience has not been addressed yet by any sustainability framework of evaluation criteria of energy technologies, however it has been highlighted as a major issue by some recent studies (Christensen et al. 2011; Ebinger and Vergara 2011; Dowling 2013).

3.2. Social category

Considering the weaknesses on the category of social indicators, the NEEDS Project aimed to target this issue through participative procedures (Hirschberg et al. 2007; Paul Scherrer Institut 2009; Carrera and Mack 2010). NEEDS and Carrera and Mack (2010) involved the establishment of a set of criteria and indicators for use in evaluation of future electricity-generating technologies with clear balance between environmental, social, and economic dimensions. Mortality and morbidity, accident fatalities, and aesthetic/functional impact have been highlighted as the most prominent social criteria. Furthermore, the level of public opposition to future plans of installation of energy technologies has been also identified as an important social issue that should be seriously taken into account during the evaluation process (Carrera and Mack 2010).

3.3. Economic category

Regarding the economic category, the following criteria for evaluation were identified by Shen, Li, and Yuan (2010) and Komor and Bazilian (2005): local economic development, increasing employment, technical maturity, potential for commercialisation, market size, and reasonableness for investment cost. Investment cost, which involves all costs related to purchase of equipment, engineering services, and technological installations, among others is another important consideration. Investment cost is a commonly used economic criterion that has been presented in many studies. Many studies also support the inclusion of job creation in the evaluation of energy projects (e.g. Haralambopoulos and Polatidis 2003; Ragwitz et al. 2005). The creation of employment opportunities is a key priority globally but also in the European context since high unemployment rates have become a key concern in many European countries and cities particularly after the financial crisis of 2009. Employment creation in few cases is included in the social dimension instead of economic by the European Commission (2003). In some cases there is a distinction between long- and short-term employment, whereas Del Rio and Burguillo (2008) and Kowalski et al. (2009) specify employment generation and creation of jobs at the local level.

3.4. Energy category

In the assessment of Shen, Li, and Yuan (2010), and as supported by other studies, energy criteria, focusing on the resilience aspect of the energy systems (Molyneaux et al. 2012), such as energy price stability, security for energy supply, low-energy prices, stability for energy generation, and peak load response (Streimiekene 2010) should be used in the evaluation of energy technologies. As the electricity sector is vulnerable to price fluctuations due to significant factors, such as production, policy matters, natural disasters, and unstable geopolitics, energy price stability should be taken into account. Security of energy supply, another important criterion, could be increased by taking advantage of local renewable energy sources (O'Brien and Hope 2010). As electric power from renewable energy can be intermittent, it is important to ensure electricity production. As such, it is also necessary to consider the stability of energy generation. Various studies (e.g. Komor and Bazilian 2005; Shaw and Peteves 2008) as mentioned in Shen et al. (2010) have also emphasised the importance of low-energy prices as it is important to maintain the standard of living of citizens.

3.5. Technological – market category

Technological maturity is also a salient consideration for evaluation as more mature technologies are expected to have high success rates (Huang, Chu, and Chiang 2008 in Shen, Li, and Yuan (2010). However, there are also technologies that are deployed in pilot sites and hence are not subject to large-scale utilisation. In some countries, policy measures enable the commercialisation of these renewable energy technologies. Hence, the potential for commercialisation has been considered in the assessment. Studies (e.g. Lee, Yoon, and Kim 2007) have underlined the significant role of potential market size in industrial competitiveness. The market size – whether domestic or international – needs evaluation; a larger market size would naturally attract investments which would facilitate industrial development.

Based on the aforementioned discussion we developed an integrated framework of sustainability and resilience indicators for the assessment of low-carbon energy technologies. Table 2 shows the list of evaluation criteria and their corresponding descriptions. Appendix 2 also provides the literature sources of these criteria.

3.6. Social validation and refinement

After completing the experts' validation phase, stakeholders' views had to be incorporated in the final set of criteria and indicators. Therefore, various stakeholders from the field of urban energy in Europe were requested to be part of the stakeholders' validation and refinement phase. As such, individual and institutional members of different associations and networks within the urban energy sector were invited. Moreover, the study was supported by the Local Governments for Sustainability, European Secretariat (ICLEI Europe) and the Intelligent Energy Europe (IEE) project, Covenant CapaCITY.

The survey respondents were asked to improve the set of evaluation criteria and indicators under investigation. Based on their local contexts and with their knowledge, expertise, and experience, the respondents were requested to add on, remove, or adjust the criteria and indicators for evaluating low-carbon energy technologies. The survey tool used for the study was accessed and completed online by the survey respondents.

The objective was to come up with a final list of criteria and indicators for utilisation in the criteria weighting survey which sought to elicit stakeholders' preferences. This activity further broadened the basis for the selected criteria and indicators for evaluating low-carbon energy technologies. The results of the stakeholders' validation established the wide acceptance of the indicator set with the range of energy stakeholders who participated in the process.

4. Results of the stakeholders' indicators validation survey

Thirty respondents from 18 European countries participated in the survey on refinement and validation of evaluation criteria and indicators. Almost half (43%) of the respondents represented Southern Europe (Italy, Greece, Spain, Portugal, and Croatia). Twenty per cent of the respondents came from Eastern Europe (Georgia, Bulgaria, Romania, and Turkey), while another 20% were from Northern Europe (UK, Denmark, Sweden, Ireland, and Lithuania). Seventeen per cent of the respondents represented Western Europe (Belgium, Austria, France, and Germany).

The survey respondents were associated with five stakeholder groups. These were: (1) civil society and non-government organisations (27%), (2) energy agencies (27%), (3) governmental organisations both local and national (23%), (4) academic/research institutes and consultants/advisors (14%), and (5) market players (i.e. electricity and energy associations, regulators and network administrators, electricity producers) (9%).

Majority of the respondents opted for the retention of all 23 criteria and indicators for evaluating low-carbon energy technologies. After the completion and analysis of the survey results on refinement and validation, there was a modification in the final selection. Two economic criteria, namely employment (short run) and employment (long run), were integrated into one as (local) employment generation. No additional criteria and indicators were added into the final selection. The number of criteria and indicators for evaluation were reduced from the original list of 23–22.

Since the majority of the respondents supported and validated the selected criteria, no significant changes were made. However, the researchers tried to integrate most of the respondents' comments and suggestions by streamlining the description of criteria. This is to highlight the scope of the study and to justify why certain criteria were included in the end. In addition, it became obvious that certain comments from the respondents were due to misinterpretation of the description of criteria and therefore the streamlining also aimed to remove all possible misinterpretations that could occur during the stage of technologies impact assessment and criteria weighting.

CO₂eq emissions and ecosystem damages were the most favoured with the majority (97%) of the respondents voting for the retention of each criterion (see Figure 3).

Figure 3. Survey results for environmental criteria.

According to the respondents, ‘noise pollution is not important in global warming’ and it is ‘difficult to assess and explain (to public) for the different technologies'; hence, the suggestion for removal. However, noise pollution is directly related to sustainability and therefore is relevant to the sustainability assessment of low-carbon energy technologies. According to Hirschberg, et al. (2007) noise pollution is considered an important criterion when evaluating energy technologies. The level of importance in any case would be elicited during the weighing process. Furthermore, noise pollution impacts have been assessed by experts (Grafakos and Flamos 2015).

Climate resilience was favoured by the 87% of the respondents, whereas one respondent stated that the criterion needed adjustment with the explanation that ‘climate change in the future might be less gradually altered’ and that ‘changes will be more sudden’. Three respondents who voted for the removal of this indicator mentioned its irrelevance ‘as nobody can predicate climate change’ and that the objective is to ‘mitigate climate change’.

The main objective of the assessment is not only to look at the climate mitigation objective but also to consider all other important sustainability aspects, such as environmental, social, and technological ones, in an integrative manner. Therefore, the resilience of the energy systems to future climate is an important aspect to be taken into account even with certain degree of uncertainty of the future climate (The Royal Academy of Engineering 2011). There are already climate models providing scientific evidence on climate predictions for the coming decades in the European continent (IPCC 2007; Christensen et al. 2011) and good indications on the likely impacts on future energy systems (Ebinger and Vergara 2011; Dowling 2013).

Regarding the criterion of (radioactive) waste 87% opted for its retention. Seven per cent of the respondents were in favour of the removal of (radioactive) waste as one explained that it is ‘not relevant’. (Radioactive) waste is an important criterion as this poses potential harmful impacts to environment and even when handled properly is still subject to human aversion. Also, 7% of the respondents were in favour of adjustment with one respondent suggesting that waste and radioactive waste should be separated because it is not one and the same. The criterion under evaluation pertains solely to radioactive waste and thus adjusted accordingly.

With regards to waste disposal (infrastructure), 90% of the respondents favoured its retention, whereas 7% of the respondents opted for its removal with one respondent explaining that it is ‘not widely acceptable’. Just one respondent was in favour of adjustment with the comment of ‘no dangerous wastes can be treated as common waste’.

As for fuel use, 90% of the respondents were in favour of keeping it as it is, whereas 10% were in favour of adjustment with the following suggestions: ‘amount of primary energy should be used instead’ and ‘fossil energy use (gas, coal, etc.)’. The notion of this criterion is to express the availability or scarcity of the fuel. Apart from fossil fuels used in gas turbine combined cycle and integrated gasification combined cycle, uranium is used as a fuel in nuclear power plants. Renewable energy sources, on the other hand, require minimum use of fossil fuels during their production phase.

Similarly, the vast majority of respondents (90%) favoured the retention of land-use requirement criterion. Three per cent of the respondents were in favour of the removal of land-use requirement as it is considered of ‘minor importance compared to the others'. Seven per cent of the respondents said that it needed adjustment as it is ‘not very important’ and ‘not only environmental criteria, [but] it is one of the most pressing social criteria as well'. However, for this research study, the methodology allows for the provision of factors of relative importance of criteria, depending on stakeholders' preferences, and therefore respondents determine the level of importance of criteria. Land-use requirement remains a relevant criterion according to different authors (e.g. Afgan, Carhalho, and Hovanov 2000; Beccali, Cellura, and Mistretta 2003; Flamos et al. 2004), and it is conventionally classified under environmental category with clear social implications as well.

Level of public resistance/opposition and accidents and fatalities were the most favoured social criteria by the vast majority of respondents (93%) (see Figure 4). However, two respondents thought that the level of public resistance/opposition needed adjustment as ‘there should be differences between resistance to nuclear or wind’. The latter comment, however, is already captured by the experts' judgements which are reflected in the impact assessment matrix.

Figure 4. Survey results for social criteria.

Eighty per cent of the respondents were in favour of keeping the criterion of aesthetic/functional impact. Ten per cent of the respondents deemed that aesthetic/functional impact needed adjustment. The respondents thought that aesthetic/functional impact ‘fits better with the environmental indicators'. Aesthetic/functional impact is an important criterion that combines both aesthetic and landscape impacts and is also related to perceptions of citizens. Therefore, it entails both social and environmental components. The research team decided to keep it in the social criteria category for a better balance between social and environmental categories.

One of the respondents who voted for the removal questioned how aesthetic/functional impact is measured. As provided in the general definition, aesthetic/functional impact is measured in the relative ordinal scale, and it was assessed by experts during the experts' impact assessment survey which is included in the impact assessment matrix (Appendix 1).

Also, two of the respondents deemed necessary to adjust the indicator mortality and morbidity. Three of the respondents, on the other hand, suggested its removal. However, mortality and morbidity were favoured by the vast majority (83%) of the respondents. Furthermore, it is certainly relevant as a criterion in the evaluation of energy technologies as it also reflects the health impacts of air pollutants. Certain pollutants, such as particulate matter, for example, are main causes for mortality and morbidity of people near power plants (Chatzimouratidis and Pilavachi 2008).

For levelised costs, 90% of the respondents opted for retaining the criterion as it is (see Figure 5). For the same criterion, two respondents (7%) opted for its adjustment with the explanations that ‘costs need to be compared with the costs of another solution and/or of not acting’ as well as the need ‘to identify costs and benefits' and ‘subsidies (for fossil fuels) and environmental cost should be included’. The comparison between the costs of technologies is already being addressed by this research study. Moreover, the MCA approach addresses the costs and benefits, including environmental-related ones (e.g. ecosystem damages, reduction of GHG emissions) of the different low-carbon energy technologies. Within the context of electricity generation, levelised costs of energy reflect the costs of building, operating, and maintaining a facility within the life cycle of the project (IEA 2010).

Figure 5. Survey results for economic criteria.

As for employment, again 90% of the respondents opted for keeping the criterion. Three respondents opted for the adjustment of this criterion, whereas one expressed the comment that ‘local jobs solution (is) less interesting if the jobs are created elsewhere'. One conventional view conveys that renewable energy generation creates additional jobs as decentralisation provides more labour-intensive employment. Moreover, it is argued that this sector puts the employment opportunities in the energy industry in domestic terrains where fossil resources are low (IEA 2012).

With regard to the energy criteria, the vast majority of respondents confirmed the selection of criteria and favoured their selection (Figure 6). The most favoured energy criteria were stability of energy generation (96%), followed by energy cost stability/sensitivity to fuel price fluctuation (93%), market concentration on supply (93%), and peak load response (90%). Three per cent of the respondents said that energy cost stability/sensitivity to fuel price fluctuation needed adjustment as ‘[to] renewable's fuel prices fluctuation is not very significant’. Energy cost stability/sensitivity to fuel price has already been studied, and the estimates were derived from expert's judgements as reflected in the impact assessment matrix. Indeed, renewables are the least sensitive to such fuel price fluctuations, whereas fossil-based fuel technologies are highly sensitive. Therefore, the criterion was included in the assessment.

Figure 6. Survey results for energy criteria.

Three respondents thought that peak load response should be removed with one respondent explaining that it ‘can be solved by smart grids or other energy production’. Smart grids would not be fully operationalised in the whole European continent. Therefore, we decided to keep this indicator in the current set and let the respondents during the weighting stage to decide on its importance, depending on the local context.

Market concentration on supply needed to be adjusted according to 3% respondents as the ‘criterion is not very significant to renewable’. The weighting elicitation that has been applied enables one to derive the relative importance of one criterion compared with another. One respondent was also in favour of its removal with the explanation that is ‘(already) included in energy cost stability’. As it has been described, the criterion of ‘energy cost stability to fuel fluctuation’ refers only to sensitivity to fuel price sensitivity and not to the power that certain suppliers may enjoy due to their oligopolistic market position since this is captured at a different criteria namely ‘market concentration’. To avoid similar misinterpretations during the weighting stage, clarifications were added on the descriptions of the ‘energy cost stability to fuel fluctuation’ and ‘market concentration’ criteria.

Most of the respondents expressed their clear preference of retaining all technological criteria (see Figure 7). The most favoured technological criterion was technological maturity (96%), followed by market size – domestic (93%), innovative ability (93%), and market size – potential export (90%). Three of the respondents thought that market size (potential export) needs to be removed, with one respondent explaining that it is ‘not relevant’. The relevance of market size – in general – and potential export in particular should not be understated as it would provide potential economic opportunities and possibly further technological advancements as well. A larger market size attracts more investments. Many countries, also in Europe, aim to expand their market shares not only domestically but internationally as well while developing local business opportunities to meet the demand (Lewis and Wiser 2007; Shen, Li, and Yuan 2010).

Figure 7. Survey results for technological criteria.

Only two respondents suggested the removal of innovative ability as it is ‘not necessary [as] we need stable production'. Stable production, as was suggested, is captured by energy category criteria. There was no additional indicator suggested by the survey respondents for inclusion under the technological criteria. Figure 8 summarises all the responses of the validation stakeholders' survey.

Figure 8. Validation survey overall results.

5. Discussion and concluding remarks

One of the main innovative aspects and contributions of the study is the integration of sustainability and resilience indicators in an overall assessment framework for low-carbon energy technologies which, to the best of our knowledge, is lacking in the literature. There are numerous studies looking at the sustainability aspects of energy technologies and few that have been conducted the last years focusing explicitly on resilience aspects (O'Brien and Hope 2010; Molyneaux et al. 2012). It is deemed necessary to develop approaches and frameworks that consider both sustainable development sub-systems and technology development in order to enhance sustainable technology development (Musango and Brent 2011). Different low-carbon energy technologies, particularly in the electricity sector, have multiple impacts with regard to environmental, social, and economic dimensions of sustainability, while at the same time are vulnerable to external shocks and disturbances due to energy prices fluctuation, concentration of energy supply, future climate change and other natural disaster threats, and reliance on non-renewable resources (O'Brien and Hope 2010; McLellan et al. 2012; Molyneaux et al. 2012). This integrated assessment framework of indicators is the first attempt to bring together sustainability and resilience aspects providing an analytical tool to policy-makers on identifying the potential sustainability impacts and vulnerabilities of different energy technologies.

During this study and the extensive literature review that was conducted, it was found that often some resilience aspects of energy systems have been considered implicitly in sustainability assessment frameworks (e.g. energy security of supply, fuel price fluctuation) (see also Appendix 1). However, other types of resilience issues (e.g. potential climate change impacts) were completely neglected. Furthermore, interestingly it was found that specific aspects could be seen either from a sustainability or resilience perspective. For instance, the indicator of ‘GHG emissions’ could be considered as an environmental impact through the contribution to GHG emissions and climate change problem but also as a resilience aspect, reflecting the financial risks of carbon-intensive technologies, in case a price is tagged on carbon either through carbon tax or a higher price of carbon emission allowance (Molyneaux et al. 2012). In that case, both perspectives should be considered explicitly by giving emphasis and possibly additional weight to these criteria.

Moreover, this framework can be adjusted and used either by local or national policy-makers for the integrated assessment of specific energy technologies. The application of the assessment framework aims to enhance guidance and evidence-based support of local and national decision-makers when planning and developing energy technologies and policies towards a low-carbon and resilient development path. By this paper we hope to further trigger discussion on the importance of explicitly integrating sustainability and resilience aspects and indicators in the assessment of low-carbon energy options, technologies, and policies.

The developed integrated assessment framework of indicators has been applied for the evaluation of selected current and future low-carbon energy technologies in Europe at the local level by incorporating stakeholders' preferences in the assessment process in order to enhance legitimacy, participation, and learning (Grafakos, Enseñado, and Flamos 2015). The framework applied with the support of the Covenant CapaCITY, a project co-funded by the IEE programme, and led by the Local Governments for Sustainability ICLEI. The majority of respondents approved the integrated framework of criteria and indicators and its application for evaluating low-carbon energy technologies (Grafakos, Enseñado, and Flamos 2015). Moreover, the same study explored synergies between sustainability and resilience aspects. For instance, the final results of different energy technologies showed that renewable energy sources contribute both to environmental sustainability and system resilience. The issue of possible interrelationships between different indicators (including sustainability and resilience indicators) was further explored in another study Grafakos, Enseñado, Flamos, and Rotmans (2015) which showed a significant positive relationship between the respondents' preferences on ‘CO₂ emissions reduction’ and ‘climate resilience’. A possible future research topic could explore the sensitivity of input variables with regard to the final ranking of the energy technologies.

Another novel aspect of the current study is the modification of the ‘3S’ validation process in the context of low-carbon energy planning and assessment. The modified ‘3S’ validation process along with the involvement of a wide range of experts and stakeholders made possible the development of a refined set of evaluation criteria and indicators.

Although there were few suggestions for the adjustment or removal of criteria, the validation process proved important as it revealed several misinterpretations of criteria descriptions that were addressed at this stage prior to weighting. The descriptions of the criteria were improved and better reflected stakeholders' suggestions and minimised possible misinterpretations of criteria during the weighting phase that might have affected the weighting results and final ranking of technologies. In addition, we could also observe that the misinterpretations were mainly expressed from few individual respondents and none from specific type of stakeholders that could have suggested removal of specific criteria. Furthermore, the validation process allowed stakeholders to express their suggestions for removal or adjustment of certain criteria. This indicated which criteria could be potentially weighted with a low level of relative importance during the weighting process.

Looking both at the results of the stakeholders' validation survey (current study) and criteria weights (study of Grafakos, Enseñado, and Flamos 2015), specifically the correlation between the percentage of removal and the ranking/weighting of criteria, aesthetic/functional impact seemed to be the less preferred – and least important – criterion. Based on the two aforementioned attributes, aesthetic/function impact was suggested for removal by 10% of the respondents (as well as for adjustment by another 10%) and has ranked 20th in the final ranking. It also ranked last (22nd) in the initial ranking of respondents. In other words, the criterion of aesthetic/functional impact was suggested for removal and also received a very low weight and ranking evaluation (Grafakos, Enseñado, and Flamos 2015).

Most of the times researchers either develop indicators intuitively or consider only experts' judgements during selection of indicators, neglecting stakeholders' perspectives (Cloquell-Ballester et al. 2006; Hak, Kovanda, and Weinzettel 2012). The proposed approach integrates stakeholders' views in the very initial stage of the assessment process, namely during the selection and validation of indicators. This could effectively reduce the risk of conflict between energy project designers and relevant stakeholders (Cloquell-Ballester et al. 2006).

The scrutiny of the criteria selection and validation process and inclusion of stakeholders (a) enhanced the relevance of criteria and indicators, (b) contributed to improved and clearly described set of criteria and indicators, (c) enhanced the decision-making process by incorporating aspects both on sustainability impacts of energy technologies such as environmental and socio-economic impacts but also on resilience and vulnerability aspects of the technologies, (d) improved the robustness of the assessment framework by increasing the acceptance of selected criteria, and (e) provided a first indication of the potentially least important criteria, (f) allowed stakeholders' active participation, while at the same time, (g) built a flexible and adaptive decision-making process that can be easily adjusted to different local circumstances.

Acknowledgements

The authors would like to thank Carsten Rothballer, Giorgia Rambelli and Maryke van Staden, energy and climate officers of ICLEI Europe, for their support during this study. In addition, the authors would like to thank all 58 European participants (stakeholders and experts in the local energy field) who were involved in the whole process from the selection and validation to the weighting of the evaluation criteria.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This research study was supported by the Covenant CapaCITY, a project co-funded by the IEE programme, and the Local Governments for Sustainability ICLEI, European Secretariat. Duration of the project: 2011–2014.

Appendix 1. The 20 most used assessment criteria for energy-related projects (Donkelaar ten and Amara 2010).

No.	Criterion	Category	Score
1	Energy and water use and savings	Environment	8
2	Raw materials use and savings	Environment	6
3	Greenhouse gas emissions	Environment	11
4	Air pollution	Environment	12
5	(Ground and surface) water pollution	Environment	10
6	Depletion of fresh water resources	Environment	8
7	Soil degradation	Environment	5
8	Waste creation and disposal	Environment	8
9	Use and management of hazardous chemicals and waste	Environment	5
10	Impact on biodiversity (or flora and fauna)	Environment	12
11	Impact on landscapes or land use	Environment	8
12	Noise	Environment	8
13	Cost efficiency	Economic	7
14	Employment creation	Economic	9
15	Health issues (mortality and morbidity)	Social	8
16	Safety issues (e.g. accident rates)	Social	5
17	Influence on food security	Social	6
18	Local income generation	Social	5
19	Education component /capacity building / awareness raising	Social	5
20	Equal opportunities	Social	5

Appendix 2. Final validated set of criteria and indicators and their literature sources.

Criteria categories	Indicators	Description	Sources
Environmental	CO2eq emissions	The indicator reflects the potential impacts of global climate change caused by emissions of GHGs for the production of 1 kWh	Afgan, Carvalho, and Hovanov (2000), Lee, Yoon, and Kim (2007), Diakoulaki and Karangelis (2007), Streimikiene et al. (2010), Burton and Hubacek (2007), Chatzimouratidis and Pilavachi (2007), Chatzimouratidis and Pilavachi (2008), Wang et al. (2009), Evans, Strezov, and Evans (2009), Jovanovic et al. (2009), Loken, Botterud, and Hoken (2009), Beccali, Cellura, and Mistretta (2003), Komor and Bazilian (2005), Shen, Li, and Yuan (2010), San Cristobal (2011)
	Climate resilience	The degree of resilience of the energy technology to the future climatic changes and extreme weather events	Ebinger and Vergara (2011), Dowling (2013)
	Noise pollution	Part of population feeling highly affected by the noise caused due to the function of the energy facility. This indicator is case sensitive and could have been measured as a factor of the noise generation by the energy technology estimated in dB multiplied by the number of people affected by the noise. However since we are investigating different energy technologies and systems at a European scale we cannot measure precisely this indicator and therefore we will use an ordinal relevant scale to measure the perceived noise	Evans, Strezov, and Evans (2009), Carrera and Mack (2010)
	(Radioactive) waste	Amount of (radioactive) waste generated by the plant divided by energy produced	NEEDS, Afgan, Carvalho, and Hovanov (2000), Streimikiene et al. (2010), Begic and Afgan (2007)
	Waste disposal (infrastructure)	Waste generation during the life cycle of the fuel and technology or availability of waste disposal infrastructure	Afgan, Carvalho, and Hovanov (2000), Streimikiene et al. (2010), Carrera and Mack (2010),
	Ecosystem damages	This criterion quantifies the impacts to flora and fauna due to acidification and eutrophication caused by pollution from the production of 1 kWh electricity by the energy system and technology	European Commission (2005), Evans, Strezov, and Evans (2009)
	Land use requirement	The land required by each power plant and technology to be installed	Afghan and Carvalho (2002), Beccali, Cellura, and Mistretta (2003), Streimikiene et al. (2010), Chatzimouratidis and Pilavachi (2008), Wang et al. (2009), Evans, Strezov, and Evans (2009), Shen, Li, and Yuan (2010)
	Fuel use	Amount of fuel use per kWh of final electricity consumption	Afgan, Carvalho, and Hovanov (2000)
Social	Level of public resistance/opposition	Energy system induced conflicts that may endanger the cohesion of society (e.g. nuclear, wind, CCS). Opposition might occur due to the perceptions of people regarding the catastrophic potential or other environmental impacts (aesthetic, odour, noise) of the energy technology/system. This indicator also integrates the aspect of participatory requirement for the application of the technology. The higher the public opposition, the higher the participatory requirement is	Tsoutsos et al. (2009), Carrera and Mack (2010),
	Aesthetic/functional impact	Part of population that perceives a functional or aesthetic impairment of the landscape area caused by the energy system. The aesthetic impairment is judged subjectively and therefore this criterion fits in the social category than the environmental one. In addition this is also a very location specific indicator and therefore an average metric will be determined measured in relative ordinal scale	Evans, Strezov, and Evans (2009), Carrera and Mack (2010)
	Mortality and Morbidity	Mortality and Morbidity due to air pollution caused by normal operation of the technology. This indicator is considered as an impact and composite indicator since it integrates all human health impacts caused from air pollutant emissions as NOx, SO2 and PM	Streimikiene et al. (2010), Carrera and Mack (2010), Afgan, Carvalho, and Hovanov (2000), Komor and Bazilian (2005), Diakoulaki and Karangelis (2007), Begic and Afgan (2007), Chatzimouratidis and Pilavachi (2009), Chatzimouratidis and Pilavachi (2008), Jovanovic et al. (2009), Shen, Li, and Yuan (2010)
	Accident fatalities	Lost of lifes of workers and public during installation and operation. Surrogate for risk aversion. This criterion partly integrates the catastrophic potential of the energy system/technology	Streimikiene et al. (2010), Chatzimouratidis and Pilavachi (2008), Carrera and Mack (2009), Afgan, Carvalho, and Hovanov (2000), Komor and Bazilian (2005), Diakoulaki and Karangelis (2007), Begic and Afgan (2007), Chatzimouratidis and Pilavachi (2009), Chatzimouratidis and Pilavachi (2008), Jovanovic et al. (2009), Shen, Li, and Yuan (2010)
Economic	Levelised costs (incl. capital, operational and maintenance, fuel costs)	Levelised costs of energy (LCOE): investment costs, operational and maintenance costs, capacity factor, efficiency, material use	Afgan, Carvalho, and Hovanov (2000), Mamlook, Akash, and Mohsen (2001), Afghan and Carvalho (2002), Liposcak et al. (2006), Diakoulaki and Karangelis (2007), Lee, Yoon, and Kim (2007), Madlener, Kowalski, and Stagl (2007), Begic and Afgan (2007), Doukas, H., B. Andreas, and J. Psarras (2007), Wang et al. (2009), Jovanovic et al. (2009), Chatzimouratidis and Pilavachi (2009), Tsoutsos et al. (2009), Shen, Li, and Yuan (2010), San Cristobal (2011)
	(Local) employment	The extent to which the application of the technology can create jobs at the investment, operation and maintenance stage. Furtheremore the criterion of employment reflects partly the extent of the impact that the technology has to the local economic development by providing jobs and generating income	Haralambopoulos and Polatidis (2003), Beccali, Cellura, and Mistretta (2003), Komor and Bazilian (2005), Erdoğmuş, Aras, and E. Koç (2006), Madlener, Kowalski, and Stagl (2007), Streimikiene et al. (2010), Doukas, H., B. Andreas, and J. Psarras (2007), Begic and Afgan (2007), Chatzimouratidis and Pilavachi (2007), Chatzimouratidis and Pilavachi (2008), Tsoutsos et al. (2009), Shen, Li, and Yuan (2010)
Energy	Energy cost stability/sensitivity to fuel price fluctuation	The sensitivity of technology costs of electricity generation to energy and fuels prices fluctuations. The fraction of fuel cost to the overall electricity generation cost	Mamlook, Akash, and Mohsen (2001), Komor and Bazilian (2005), Liposcak et al. (2006), Begic and Afgan (2007), Streimikiene et al. (2010), Wang et al. (2009), Jovanovic et al. (2009), Shen, Li, and Yuan (2010), Komor and Bazilian (2005), Burton and Hubacek (2007), Lund (2007), Cai et al. (2009), Shen, Li, and Yuan (2010)
	Stability of energy generation	Stability of output of electric power generated depending on the technology used. This reflects whether the energy supply is being interrupted. The presence of these interruptions impacts the electricity network stability. This criterion reflects whether the energy supply faces any interruptions due to the type of energy technology	Gross (2004), European Commission (2005), Taljan and Gubina (2009), Georgilakis and Katsigiannis (2009), Tsoutsos et al. (2009), Shen, Li, and Yuan (2010)
	Peak load response	Technology specific ability to respond swiftly to large variation of demand in time / % representing the possibility to satisfy the required load	Streimikiene et al. (2010), European Commission (2005)
	Market concentration on supply	The market concentration on the supply of primary sources of energy that could lead to disruption due to economic or political reasons	Carrera and Mack (2010)
Technological/market	Technological maturity	The extent to which the technology is technically mature. The criterion refers to the level of technology's technological development and furthermore the spread of the technology at the market	Beccali, Cellura, and Mistretta (2003), Huang, Chu, and Chiang (2008), Wang et al. (2009), Tsoutsos et al. (2009), Shen, Li, and Yuan (2010)
	Market size (domestic)	Demand for final products (of energy technologies) and potential market size domestically. The potential market size plays an important role to establish industrial competitiveness and stimulate economic growth	Lee, Yoon, and Kim (2007), Lewis and Wiser (2007), Lund (2009), Shen, Li, and Yuan (2010)
	Market size (potential export)	Demand for final products (of energy technologies) and potential market size internationally	Lee, Yoon, and Kim (2007), Lewis and Wiser (2007), Lund (2009), Shen, Li, and Yuan (2010)
	Innovative ability	Flexibility and potential of the technology to integrate technological innovations	Carrera and Mack (2010)