Towards evaluation design for smart city development

Abstract

Smart city developments integrate digital, human and physical systems in the built environment. With growing urbanization and widespread developments, identifying suitable evaluation methodologies is important. Case-study research across five UK cities ‒ Birmingham, Bristol, Manchester, Milton Keynes and Peterborough ‒ revealed that city evaluation approaches were principally project-focused with city-level evaluation plans at early stages. Key challenges centred on selecting suitable evaluation methodologies to evidence urban value and outcomes, addressing city authority requirements. Recommendations for evaluation design draw on urban studies and measurement frameworks, capitalizing on big data opportunities and developing appropriate, valid, credible integrative approaches across projects, programmes and city-level developments.

Introduction

Rapid urbanization of city regions creates complex pressures on infrastructure, systems and services, as well as citizens and the environment, triggering the need for innovative, sustainable solutions to urban development challenges (Caragliu, Del Bo, and Nijkamp 2011). Urban design and planning have an ongoing role in addressing urban challenges through improving ‘…connections between people and places, movement and the urban form, nature and the built fabric’ (Radford 2010, 380). Smart approaches, defined by the British Standards Institution (BSI) as ‘the application of autonomous or semi-autonomous technology systems’ (BSI 2014a, 12), can contribute to urban solutions, particularly through the establishment of digital connections between networks (telecommunications, satellite communications and the Cloud), sensors (embedded sensors and actuators, proximal and remote sensing) and interconnected Information and Communication Technology (ICT) devices, known as the Internet of Things. However, smart city development is essentially a multi-disciplinary endeavour rather than simply offering a technological fix for urban challenges. It requires the ‘effective integration of physical, digital and human systems in the built environment to deliver a sustainable, prosperous and inclusive future for its citizens’ (BSI 2014a, 12).

Despite widespread smart city developments across Europe and other continents, strikingly little research has been conducted on the evaluation of smart city interventions and the measurement of outcomes of embedded smart technologies for cities and citizens (Bis 2013; EU Directorate-General 2014). Evaluation practice has been limited by a lack of clarity concerning definitions and the best approaches to measure the contribution of smart solutions to city performance (BSI 2014b). The European Innovation Partnership on Smart Cities and Communities (EIP-SCC) observed that there are currently no standardized smart city indicator frameworks, widely-accepted by cities to measure city performance and evaluate progress against urban strategies (EIP-SCC 2013). A big challenge therefore is to determine the value of smart urban developments and to evidence the impacts on city outcomes. This paper builds on findings from the SmartDframe research linked to the MK:Smart programme (mksmart.org), which examined city approaches to evaluation and reporting of smart city developments and their impacts on city outcomes, through five UK city case studies (Caird, Hudson, and Kortuem 2016). First, key conceptual, measurement and evaluation issues in smart city evaluation are examined through reviews of the academic and corporate literature. Second, case studies of city approaches to evaluation of smart city programmes and projects in Birmingham, Bristol, Manchester, Milton Keynes and Peterborough are described with reference to evaluation practices, challenges and city authorities’ recommendations. The paper concludes with consideration of how to draw on existing measurement and evaluation work to support the design of appropriate, effective and credible evaluation of smart urban developments, and to identify their outcomes for cities and citizens.

The smart city concept

It has been asserted that the smart city is the most popular, worldwide vision for a successful future city, and is even more popular than, for example, visions of liveable cities, inclusive cities, innovative cities, digital cities, sustainable cities, green cities and so on (Moir, Moonen, and Clark 2014). While there are many definitions of smart cities available (Albino, Berardi, and Dangelico 2015), there are several key characteristics.

(1)	The smart city has integrated ICT infrastructure and technologies (BSI 2014c) for improving city functioning (Hollands 2008) and achieving the digital transformation of urban systems.
(2)	One characteristic focuses on the development of human capital (Hollands 2008; Caragliu, Del Bo, and Nijkamp 2011) through ICT-enhanced governance to support sustainable urban development driven by the knowledge, creativity, innovation and entrepreneurship of city actors (Hollands 2008).
(3)	Central to the smart city concept is big data, which describes information/data assets characterized by high ‘volume’, ‘velocity’, ‘variety’, ‘variability’ and ‘value’ for different stakeholders that requires high-capacity cloud-processing services (Fujitsu 2012). The data assets include real-time and near real-time data-streams from city infrastructures and sensor networks, and contemporary space platform technologies and web services. It includes volunteered and crowd-sourced citizen data from social media, mobile apps and citizen participation platforms, and also traditional static historic/legacy data sources collected through surveys. Corresponding predictive analytics, machine learning and stream statistics are required to mine and reveal patterns in voluminous, fast-changing, diverse, structured and unstructured data sources to develop data intelligence. In this way, big data enables the development of data-driven urban systems and services (Bis 2013).
(4)	One characteristic addresses the development phases of smart cities (Zygiaris 2013) and their smart development maturity (IDC 2013). The smart city label applies both to newbuild cities, where Greenfield city developments are most commonly recognized as smart cities, for example, Masdar (UEA) and Songdo (S. Korea), and also to cities with a variety of retrofit smart city developments, for example, Rio de Janeiro and Barcelona (Batty et al. 2012; Shelton, Zook, and Wiig 2015).
(5)	A diverse range of smart city projects underpin smart city development, implemented at different scales across buildings, neighbourhoods, cities and regions (Bosch et al. 2017). These projects address a broad range of city challenges through urban regeneration (e.g. carbon-neutral smart neighbourhood infrastructures), urban development (e.g. smart energy, water and waste resource management systems and smart grids) and urban innovation (e.g. testbed micro-infrastructures for networks, intelligent traffic systems, open-data and citizen co-creation platforms) (Batty et al. 2012; EU Directorate-General 2014).

The smart city vision is typically presented as beneficial in both ambition and outcomes. The BSI smart city vision is citizen-centred, collaborative, digital and characterized by open data (BSI 2014c). This is associated with extensive social, economic and environmental benefits that maintain or improve quality of life for citizens (BSI 2014c). However, the merits of the techno-utopian smart city paradigm have been also contested from dystopian perspectives on smart cities (Hollands 2008; Townsend 2013; Kitchin 2014; Vanolo 2014). For cities, there are concerns about ‘new geometries of power’ (Vanolo 2014, 884), including: the political and corporate use of big data; technocratic governance; corporate dominance of city systems; technological lock-ins; the security of hackable, attackable city systems (Kitchin 2014); panoptic surveillance and control of citizens (Townsend 2013; Kitchin 2014); and public-sector marginalization through public-private city partnerships (Vanolo 2014). For citizens, there are additional issues of data privacy and control (Kitchin 2014), and concerns about social inequalities and marginalization concerning access to the benefits of smart cities (Hollands 2008). With little research conducted on the evaluation of smart city developments (Bis 2013), there is a need for more evidence of the benefits and disbenefits for cities and citizens.

Standardization and smart urban metrics

Considerable international and national work is presently ongoing through the International Standards Organization (ISO), European Committee for Standardization (CEN) and BSI to establish good standards in smart urban development and performance metrics (SSCC-CG 2015). The ISO has already agreed standards for ‘Smart Community Infrastructures’ performance metrics (for example, ISO/TR 37150:2014 and ISO/TS 37151:2015) (see iso.org). BSI, working with ISO, have provided a significant body of work on smart city standards and urban performance metrics (BSI 2014a, 2014b, 2014c), including the Smart City Framework, Publicly Available Specification PAS181 (BSI 2014c). Moreover, a key European Commission (EC) EUROCITIES initiative called CITYkeys (citykeys-project.eu) aims to develop valid city performance measurement frameworks, Key Performance Indicators (KPIs) and standardized data collection procedures to speed up the diffusion of smart city solutions across cities through supporting comparable, scalable and replicable smart city solutions (Bosch et al. 2017).

Standardized smart urban indicators and metrics are not widely accepted by cities while the development of standards is at early stages. Recommendations by the European Innovation Partnership on Smart Cities and Communities (EIP-SCC) are that standardized smart city measurement indicators should:

•	Align with city strategies and operational levels of city development;
•	Establish measurement over-time, evidenced against baseline measures and strategic targets, using mainly real-time data;
•	Develop through a stakeholder process with expert and community groups;
•	Be open to improvement and future innovation;
•	Build on existing urban development and performance indicators aligned with typologies of European cities;
•	Support open reporting and cities’ evaluation of progress towards becoming smart through comparative city benchmarking (EIP-SCC 2013).

Smart city indicator frameworks and city indexes

Extensive reviews of city indexes (such as conducted by Moonen and Clark 2013; Albino, Berardi, and Dangelico 2015; Joss et al. 2015) have identified surprisingly few published smart city indicator frameworks designed to measure smart city development and performance outcomes. This selective review identifies two models providing insight into the phases and indicators of smart city development (Table 1), and a further five smart city indicator frameworks and city performance indexes (Table 2).

(1)	The Smart City Maturity (SCM) Model recognizes that cities are at different maturity phases of becoming smart. This offers a high-level, city benchmarking, comparative tool to identify the maturity phases of a city’s smart city development based on smart city strategies and the scale of development. The maturity phases range from the least mature ad hoc project-planning phase to the most mature smart city ‘Optimized’ phase characterized by a city-wide city of systems (IDC 2013).
(2)	The Smart City Reference (SCR) Model was designed to identify innovation policies and processes needed to support planning for smart, sustainable urban development. This offers an in-depth conceptualization of the range of work needed for smart city development through seven city layers/stages, including integrated ICT infrastructure and technologies to support capabilities for city intelligence and innovation services (Zygiaris 2013). This has similarities to IBM’s Smarter City Assessment Tool for identifying cities’ capabilities for smart urban development through interconnection, instrumentation and intelligence (IBM plc 2009). Each city layer is associated with different types of development, and different Key Performance Indicators (KPI) linked to the performance, economy, efficiency and effectiveness of new city infrastructure, systems and services (Zygiaris 2013).
(3)	The European Smart Cities Ranking (ESCR) Model (smart-cities.eu) aims to measure medium-sized European cities against smart city indicators to enable city ranking, benchmarking and inter-city comparisons (Giffinger et al. 2007). The ESCR Model offers a comprehensive smart city indicator framework defined across six city characteristics/dimensions, namely Smart Governance, Economy, People, Living, Environment and Mobility, and includes both development and performance indicators, building on data collected at local, regional and national spatial levels across European countries (Giffinger et al. 2007).
(4)	The Smart City Index Master Indicators (SCIMI) framework is a Smart Cities Council initiative to enable ranking cities in terms of liveability, workability and sustainability indicators (smartcitiescouncil.com/resources/smart-city-index-master-indicators-survey). SCIMI measures similar smart city dimensions to the ESCR Model, namely Smart Government, Economy, People, Living, Environment and Mobility dimensions, although it presents an alternative conceptualization, drawing on a wide range of international data sources applicable to cities and buildings (Cohen 2014).
(5)	The CITYkeys Indicator Framework measures smart city project-level success outcomes linked to smart city-level indicators across People, Planet, Prosperity, Governance and Propagation themes aligned with EU policies. Building mainly on existing urban frameworks, the People, Planet and Prosperity indicators correspond to the Triple Bottom Line of social, environmental and economic sustainability. The Governance indicators pertain to leadership in smart city developments, while the Propagation indicators identify the potential for upscaling and replication of smart city solutions in other contexts.
(6)	The Ericsson Networked Society City (ENSC) Index, developed by Ericsson Ltd with Sweco Ltd, measures the ICT maturity of networked cities against indicators of ICT Infrastructure, Readiness and Usage indicators, which correspond with the development, diffusion and adoption of ICT infrastructure and technologies. This adopts a systems approach to examine relationships between city ICT maturity across Economic, Social and Environmental Impact city dimensions (the sustainability Triple Bottom Line) (Ericsson 2014).
(7)	The Cities of Opportunity (CoO) Index of Leading Cities measures Smart, Quality of Life and Economic Indicators, drawing on extensive city indexes and open data sources (PricewaterhouseCoopers/Partnership for New York City (PwC/PNYC) 2014). This is one of the few general city indexes that measures ‘Smart’ indicators as an embedded dimension of cities using measures of Intellectual Capital and Innovation, Technological Readiness and City Gateway indicators (Table 2).

Table 1. Smart City (SC) development models.

	Smart City Maturity (SCM) Model	Smart City Reference (SCR) Model
Developer	International Data Corporation (IDC 2013)	Zygiaris (2013)
Approach	Identifies 5 SC maturity phases & indicators Supports city planning, development, city benchmarking	In-depth conceptualization of SC layers measured against KPIs Supports city planning & sustainable development
Measured city dimensions and indicators	Measures SC maturity indicators linked to 5 maturity phases:	Measures KPIs corresponding to SC layers:
	1. ad hoc project planning 2. ‘Opportunistic’ phase with proactive project deployments and emerging collaborative partnerships and strategies 3. ‘Repeatable’ projects phase with process implementation, stakeholder buy-in and formulated strategies 4. ‘Managed’ phase with formal systems for work/data flows, and technology and standards driving performance management and outcomes 5. ‘Optimized’ phase with a sustainable city-wide platform within the city system of systems	0. ‘The City’ includes traditional city components, e.g. infrastructure, networks, built environment and districts measured by city readiness to adopt smart features 1. ‘Green City’ eco-policies and planning measured by urban CO^2 footprint (emissions) 2. ’Interconnection’ with city-wide broadband ICT-infrastructure using Wi-Fi, Wi-Max, 3G+, Ethernet, Fibre, Broadband-over-Power-Lines and radio-communications technologies measured by the economy of broadband uptake (city-wide costs per metre^2) 3. ‘Instrumentation’ with real-time connections infrastructure, using wireless sensor and actuator networks, radio frequency transmitters, traffic signals, smart meters, radio-frequency identification (RFID) and the Internet of Things. Measures include real-time events/system response 4. ‘Open integration’ providing a smart environment for open and distributed information storage on technological platforms, supporting data representation, visualization, exchange-across-sectors, and data-sharing services measured by effective integration and control of smart city applications, and open resources for open integrated space. This uses the Cloud, Application Programming Interface (API), the semantic web and ontologies, and Web services. Measures are effective integration and control of smart city applications and open resources 5. ‘Applications’ add value to city intelligent services, supporting government, efficient energy use etc. Aims to measure real-life intelligence 6. ‘Innovation’ covers new business models for economic growth, new governance structures and living labs addressing quality of life, using technologies such as the Web-of-Trust (WoT) measured by smart growth, including efficiency of public infrastructure and systems, business metrics etc

Table 2. Smart City (SC) models, measurement frameworks and indexes.

Tables 1 and 2 identify a wide range of smart urban indicators focused on urban development (Table 1) and urban performance outcomes (Table 2), applicable across dimensions of governance, society, economy and environment. A comparison of Tables 1 and 2 city models and frameworks reveals different conceptualizations of the key smart city dimensions and indicators, raising challenges concerning the validity, measurability, complexity, credibility and utility of smart city indicators representing complex urban systems.

Validity issues are illustrated through a comparison of the ESCR Model (Giffinger et al. 2007) and the SCIMI framework (Cohen 2014), which both measure similar city dimensions, although based on different factors, indicators and metrics. The ESCR Model Environment indicators focus on the natural environment, whereas the SCIMI framework focuses more strongly on urban planning and the built environment. Moreover, although both have Smart Mobility indicators, the ESCR Model offers only one ICT-related factor, namely the ‘Availability of ICT- Infrastructure’, whereas the SCIMI framework offers a large number of ICT indicators. Raising further conceptual issues, the SCIMI’s indicators focus on the integration of ICT infrastructure, systems and services across city dimensions, whereas the ENSC Index is focused on indicators of development, diffusion and adoption of ICT infrastructure and technologies in networked cities.

Measurability challenges are illustrated by the ESCR Model’s difficulties in providing measures for the Smart Governance Participation factor ‘Political strategies and perspectives’ (Giffinger et al. 2007). In addition to difficult-to-measure indicators, more consideration needs to be given to intangible and tangible indicators (Carmona et al. 2001; Holman 2009). Related to intangibles, few frameworks (Tables 1 and 2) include indicators of citizen outcomes. Exceptions include the ESCR Model and CITYkeys Indicator Framework, which include citizen satisfaction indicators, for example, the ESCR Model measures satisfaction with governance, living, the environment and mobility in the city (Table 2).

Holman (2009) noted that the drive towards comprehensiveness has created large quantities of urban indicators, although the potential for comprehensive measurement is an illusion. For example, the ESCR Model includes 74 total indicators and numerous metrics, only two of which were ICT-related indicators and hence not comprehensive (Table 2). A challenge for urban indicator frameworks is to represent: the complexity of dynamic, evolving, open and unbounded urban systems (Arnold 2004); the interrelationships between slow-changing urban forms, including buildings, physical infrastructure and planned space, and relatively fast-changing flows, including capital, people, communications, energy and pollutants (Williams 2014); and the interacting social, economic, political, technological and environmental factors in urban systems shaped by virtuous and vicious system feedbacks, cumulative causation and historical path-dependencies (Arnold 2004). Systems approaches are illustrated in both the ENSC Index and CITYkeys Indicator Frameworks, although both approaches simplify cause-effect relationships in smart urban systems, focusing less on system dynamics and more on impacts (Table 2).

Approaches to developing city frameworks are intrinsically value-laden and therefore ideological, which is evidenced by the selection of indicators and measures, and decisions on data normalization, weighting systems and aggregation methods. Constructivist perspectives draw attention to the different meanings that value holds for different stakeholders (Carmona, De Magalhães, and Edwards 2002), which have many roles in shaping, implementing and engaging with smart city programmes (see BSI 2014c, 19). To be credible to expert and community stakeholders, different values may need to be reflected, both in the development and subsequent application of urban indicator frameworks.

The main utility for smart city models and frameworks (Table 1 and 2) is to support planning, city benchmarking and intercity comparisons. However, Vanolo (2014) criticized city frameworks as a simplistic, reductionist performance technology used to benchmark cities based on their city rankings. The challenge is to develop appropriate, valid, credible and useful approaches to city measurement. Holman (2009) argues that indicators should be policy instruments designed to have clear links to policy changes and innovative local governance.

Evaluation of smart city developments

Without evaluation to determine the appropriateness, effectiveness and value of programmes and projects and their impacts in specific contexts, it is difficult to judge success (Arnold 2004). In their report on international smart city case studies, Bis (2013) criticize existing approaches to smart city evaluation as inadequate and non-standard, and more focused on implementation processes and investment metrics than on city outcomes, although there were some exceptions, for example, examining citizen value in Rio de Janeiro and city benefits in Boston (Bis 2013). The major European ‘Mapping Smart Cities’ study has also conducted evaluations based on types of projects, their goals, scale, scalability, targeted stakeholders, level of citizen-engagement and success outcomes (EU Directorate-General 2014). However, despite the proliferation of smart city developments, there have been few evaluation studies.

A growing body of studies has explored evaluation and value in urban design (Carmona et al. 2001; Carmona, De Magalhães, and Edwards 2002; Chiaradia, Sieh, and Plimmer 2017), urban planning (Oliveira and Pinho 2010) urban regeneration (Tyler et al. 2013) and innovation and technology policy studies (Arnold 2004; Edler et al. 2012 ). The UK Government’s Green Book provides guidance on project and programme evaluation, including techniques to assess the economic, financial, social and environmental benefits and costs of urban design, development and innovation, with attention to different stakeholder interests (HM Treasury 2013).

The evaluation process requires setting objectives, identifying target groups, articulating key questions, clarifying impact dimensions, setting up the evaluation logistics and timing. The process requires choosing methods for data collection, assessment and analysis and judging the quality, usefulness and consequences of the evaluation (Edler et al. 2012). Consideration of urban frameworks and city indexes may offer useful measurable indicators and metrics. Assessments may draw on quantitative data-driven methods (e.g. cost-benefit, cost-effectiveness analysis, before-after assessments, audits and technical monitoring) and qualitative methods (e.g. citizen surveys, expert and community stakeholder workshops, Design Quality Indicators, Goals-Achievement Matrix, Multi-criteria analysis) (HM Treasury 2013). It is recognized that qualitative methods can capture intangibles and externalities (Carmona et al. 2001) and the different meaning of value for different stakeholder groups (Carmona, De Magalhães, and Edwards 2002).

Articulating a theory of change is a popular way to begin urban evaluations. This describes the way policy programme objectives should deliver the anticipated outcomes through a logical framework of linked activities, outputs, outcomes and impacts, with recognition of direct/indirect effects and contextual factors (Tyler et al. 2013). This logical framework may be informed by critical strategic and operational success factors for smart city development programmes, which include the establishment of a ‘benefit realization’ strategy to map, track and deliver success outcomes (BSI 2014c). Evaluation design and practices need to test the underlying programme theories (Pawson and Tilley 2004), which in smart city studies is the theory that the expected city benefits will be delivered through smart city developments.

Overview of city case studies

Emerging from this review are questions concerning how cities approach the evaluation of smart city projects and programmes. This is examined through case studies in UK cities, including Birmingham, Bristol, Manchester, Milton Keynes and Peterborough (Caird, Hudson, and Kortuem 2016; Caird 2018). Case studies were developed during 2015, following interviews with representative city authorities, and reviews of their strategic future city and smart city programmes. Cities were selected to represent different-sized conurbations, relevant to their population and geographical area. Each selected city was actively developing as a smarter city, having several funded projects within their strategic future and smart city programmes, and an involvement in UK and European smart city networks, including the EIP-SCC, Small Giants and UK Core Cities and the EUROCITIES network of over 140 major European Cities (Figure 1).

Figure 1. Overview of UK smart city case studies.

Case study: Birmingham

Smart city approach: Birmingham Council established a Smart City Commission, which includes leading figures from business, universities and the public sector supported by Digital Birmingham, the city’s digital partnership. The Commission’s Smart City Vision and Roadmap is focused on thematic areas to address economic growth and city challenges.

Projects: The ‘Technology and Place’ theme addresses issues of connectivity, digital infrastructure, open data and information markets, with projects focused on the provision of high speed broadband connectivity, free Wi-Fi in public buildings and open data platform, linked to Birmingham’s Big Data Corridor and Data Factory Portal (Figure 2). The ‘People’ theme addresses digital inclusion, citizens’ skills, employment and digital innovation with digital inclusion programmes for citizens and communities and the Smart City blog for community ideas-sharing. The ‘Economy’ theme addresses a range of issues including health and wellbeing, ICT, effective mobility, energy efficiency and carbon reduction. Several projects with the National Health Service (NHS) address telehealth services providing remote health monitoring, consultation and diagnosis. Additional ‘economy’ projects include smart traffic control and parking, intelligent energy saving, smart street lighting and an SME digital academy programme.

Figure 2. View of Birmingham City Town Hall, one of some 200 city public buildings having free Wi-Fi provided as a means of encouraging citizens to engage with city data feeds and smart city solutions.

Approach to evaluation: The Smart City Commission decided not to measure specific smart KPIs at the Roadmap level, but instead build an evaluation framework supported by PAS181 (BSI 2014a) to identify how smart city developments deliver desirable city outcomes. The EC has also been a significant influence on their evaluation approach, requiring smart city projects to provide clear measurement and input data to other EC-funded projects. Although Digital Birmingham authorities were considering ways to evaluate progress with their Roadmap actions, their primary focus has been on operationalizing projects. Their current approach was to measure progress at the project-level and work with partners to measure project KPIs. Their plan is to develop their evaluation work with partners, once they achieve greater maturity with the Roadmap deliverables.

Case study: Bristol

Smart city approach: Smart City Bristol was established by Bristol Council as a collaborative programme between different sectors, the community and citizens. It is led by Bristol Futures, a Council Directorate, whose vision is to ensure that Bristol becomes a resilient, sustainable, prosperous, inclusive and liveable place. The work is delivered through Connecting Bristol, the city’s digital partnership.

Projects: Projects were initially developed around themes of Smart Data, Transport and Energy, although the focus has expanded into new areas, including telehealth projects to address connected health information and provide digital home health assistance. Bristol’s two flagship projects include the Bristol Future City Demonstrator, which supports digital infrastructure development and the Bristol Living Lab. A second flagship project, Bristol is Open, is a joint venture with University of Bristol to provide an open digital infrastructure to develop the Internet of Things, make city performance data available through an open data portal, and test solutions relevant to city challenges, for example, traffic congestion (Figure 3). The city’s many smart transport and energy projects include driverless car trials (the Venturer project www.venturer-cars.com), intelligent energy saving in public buildings and social housing, and one of the first Council-owned smart energy companies (bristol-energy.co.uk). Bristol also works with international partners through the EU-China Smart Cities Project to engage citizens with a co-creation process to design and improve city services.

Figure 3. View across Bristol, centred on the ‘We The Curious’ collaborative science museum (previously ‘At-Bristol’) (www.wethecurious.org), with its planetarium and Open Data Dome in the foreground. The Dome can portray a variety of city datascapes aided by local city companies such as Zubr (zubr.co) and enables interactive public participation.

Approach to evaluation: Bristol’s smart city approach began with work commissioned by Bristol City Council to benchmark Bristol’s activities against international cities, leading to the Smart City Benchmark and Smart City Reports. Their approach to evaluation has been influenced by methodologies developed by the EC, and their own work to develop KPIs particularly for smart energy projects. Their evaluation work is typically project-focused, led by their project partners and driven by their funders’ requirements for evaluation of project impacts linked to KPIs. Going forward, they recognize that their smart city work needs to contribute to city strategies and challenges, although nothing specific has been established to evaluate the overall city impacts of smart city development.

Case study: Manchester

Smart city approach: The Council-led Smarter City Programme explores ways to use new technologies to optimize city systems and outcomes for citizens, focusing on themes of how and where we live, work, play, move, learn and organize in urban environments (manchester.gov.uk/smartercity/). This urban transformation programme was established within the Community Strategy framework, which is delivered through the Manchester Partnership of public, private and third sector organizations.

Projects: The Council have over 30–40 smart city projects, funded through European, national and local investment. Many more smart city projects exist, although not all are led by the Council. A major EC-funded infrastructure project, Triangulum, aims to transform the Manchester Corridor into a low carbon smart city district through establishing an electric vehicle transport infrastructure, renovating historical buildings and developing an autonomous energy grid to supply district heating and electricity (triangulum-project.eu) (Figure 4). Another key UK-funded smart city project, CityVerve, demonstrates the potential of Internet of Things technologies to engage citizens and communities in areas of healthcare, energy and environment, transport and culture (cityverve.org.uk). Manchester Council also works with international partners to improve public service efficiencies through the EU-China Smart Cities Project.

Figure 4. View across Manchester of Oxford Road, a focal point for the Corridor Manchester, Innovation District identified for urban transformation through the Triangulum project.

Approach to evaluation: Manchester’s evaluation work was at early stages with the main pressures for evaluation arising from their project funders’ requirements. The authorities did not believe that any city could claim to have a fully co-ordinated evaluation programme. The EC has been the main influence on Manchester’s evaluation work, particularly through the Council’s involvement with the EUROCITIES Smart City Forum and the CITYkeys initiative to compare Manchester’s smart city solutions with other European cities.

The Council plan to develop an Impact Assessment Framework for the Triangulum project, with University of Manchester taking a leading role. The aim is to monitor how well smart city developments work, supported by the development of a digital map of key city transport, energy and utilities infrastructures. Once established, this Framework has the potential to be expanded geographically to city-scale and linked thematically to city strategies and performance measures. However, there was concern that a top-down evaluation programme is not always appropriate and good ideas could be terminated by premature evaluation of city innovation projects at early development maturity stages.

Case study: Milton Keynes

Smart city approach: Milton Keynes smart city work is conducted through the Council-led Future City Programme, which is designed through collaborations between business, universities and government partners, including four national Catapult innovation centres. The programme aims to support economic growth, address infrastructure challenges, improve citizens’ lives and develop Milton Keynes’ reputation as a new city.

Projects: MK:Smart is one of the city’s flagship programmes, led by The Open University with partners from local government, industry and universities. This aims to develop innovative solutions in smart transport, energy, water, enterprise, citizen engagement and education. Central to the programme is the MK Data Hub, which draws together data relevant to city functions, including data from key city infrastructures, sensor networks, satellite data and social media. The Future City Programme includes the MK Internet of Things network demonstrator and several smart mobility projects, including the Electric Bus Trial, LUTZ Pathfinder (ori.ox.ac.uk/projects/lutz-self-driving-pods) and UK Autodrive (ukautodrive.com) for trialling autonomous vehicles (Figure 5).

Figure 5. View of Milton Keynes, north-east from the railway station towards the city centre. Plans with LUTZ Pathfinder are to use this route for an autonomous passenger transport service, connecting the station with the business district.

Approach to evaluation: Most of Milton Keynes’ smart city work has been externally-funded, therefore each project has reporting requirements set by respective funding bodies. The Council has not yet established an overall city-level framework for tracking the progress of the Future City Programme, although some projects have established KPIs for measuring outcomes, and some early stage projects are evaluated/judged through demonstration of an innovation concept. Since most of their smart city projects are not implemented at the city scale, it is not possible to measure impacts at the city-level. However, the Council authorities recognize difficulties in identifying the causal relationships between projects and city outcomes, and considered that evaluation may work best at the project-level where outputs are more immediately demonstrable.

Case study: Peterborough

Smart city approach: Peterborough’s Smart City work has developed through Peterborough DNA, a Future Cities Demonstrator programme, funded by Innovate UK. This is led and delivered by the City Council and Opportunity Peterborough, the city’s economic development company.

Projects: Peterborough DNA aims to address city challenges through citizen-centred projects in thematic areas focused on ‘Skills for our future’, ‘Innovation’ through citizens and entrepreneurial activity, ‘Open data’ available through a living data portal, and ‘Smart business’ with a digital platform developed to encourage business engagement with the sharing economy at the SME business area, known as Smart Fengate (Figure 6). As part of Peterborough DNA, the Council also organized a Smart City Leadership event to engage public and private sector organizations across city areas.

Figure 6. View across Peterborough, past the Cathedral, towards the Smart Fengate Business Cluster (www.futurepeterborough.com/project/smart-fengate), an industrial area of the city, where a sharing economy digital platform is being trialled to provide solutions for greater resource efficiency and reduced waste.

Approach to evaluation: When Peterborough embarked on the Peterborough DNA programme, their funders were more interested in city innovation demonstrations that could be scaled up to work in bigger cities than evaluation of the programme impacts. While BSI’s smart city standards has been a strong influence on their smart city leadership, work on evaluation was still at initial stages of development. However, the city was beginning to consider a more formulated impact assessment and they have conducted an initial evaluation to improve the DNA programme, reduce project complexity and address the potential scalability of projects. Going forward, Peterborough plans to focus on city challenges and map the key metrics and data sources available to use in smart city assessments and evaluation.

Evaluation practices, challenges and recommendations

The cities’ smart city work is embedded in their future and smart city programmes led by Councils and Directorates. This covers a wide range of smart city projects designed to address city challenges relevant to key city infrastructures and dimensions of governance. This work is a product of collaborative partnerships with public, private and third sector organizations and citizen groups. All five cities describe their future city vision as smart and/or connected, although Birmingham is the only city with an established Smart City Commission, Vision and Roadmap.

The cities’ approach to smart city evaluation is currently focused at a project level and primarily driven by their funders’ requirements. Several cities, including Birmingham, Bristol and Milton Keynes, have established project KPIs with their partners, which potentially cover a range of technical, social, economic and environmental performance measures. The city authorities considered establishing baseline measures and strategic targets and KPIs to be a good approach for monitoring performance and measuring progress over-time (as recommended EIP-SCC 2013; BSI 2014c). This would demonstrate the validity of innovation concepts and help identify projects with the biggest city impacts and replication potential.

Some of the city projects were already delivering significant data outputs aligned with city strategies, such as on energy, climate change, transport, waste, economic development and liveability. This was supporting city interests in developing data intelligence through establishing new mechanisms for generating, collecting and sharing data. All the cities operate web-based, open data portals linked to their data hubs, providing public access to information. Such data sources may be aggregated, as for example by the ESRI Inc ‘Urban Observatory’ (www.urbanobservatory.org), which permits side-by-side comparison of key data metrics from multiple cities, so visualizing the complex, urban themes of international cities across work, movement, people, public services and systems (Figure 7).

Figure 7. The ESRI Inc. ‘Urban Observatory’ showing city housing density data for Peterborough, Manchester and Bristol (www.urbanobservatory.org).

The cities were aware of work ongoing with standardization initiatives, and several were actively engaged with BSI’s standards and the CITYkeys initiatives. However, city authorities were less familiar with the smart city indicator frameworks reviewed in Tables 1 and 2, although some were concerned that these were too general when a perceived better approach would be to focus measurement on specific areas. For example, Birmingham has trialled a smart city framework specifically focused on energy, in partnership with Arup.

The case studies showed that the cities were at the early stages of developing plans to evaluate the city-level impacts of smart city developments and were working in partnerships, mainly with local universities, to address evaluation challenges. Although most were not advanced with evaluation plans, Birmingham had made progress in developing a city-level evaluation framework, aligned with their smart city strategy and Roadmap. Manchester was developing an Impact Assessment Framework for their Triangulum project and a plan to assess city-level impacts. Other cities, including Milton Keynes and Peterborough, had developed many measures through their city programmes to contribute to a smart city evaluation framework, although this work was at an early stage. However, the cities’ evaluation practices were not embedded in city management structures and performance reporting processes. Moreover, there is currently no statutory obligation for UK cities to report their smart city work through city performance and political reporting processes, and therefore the smart city work was only beginning to influence city decisions, particularly around development and investment decisions.

Some cities were unconvinced of the need for an overarching, standardized smart city framework, which might not be sufficiently relevant to their unique city challenges, strategies, circumstances and projects. Moreover, cities already have statutory obligations to measure and report numerous KPIs against city strategies and actions. For example, Bristol authorities mentioned that there are approximately 150 KPIs that the Council report on annually, which they considered burdensome. Rather than developing new smart city KPIs, some city authorities would prefer to measure the contribution of smart city projects and programmes against existing city KPIs in establishing city-level impacts.

The main evaluation challenges identified by cities centred on choosing suitable methodologies to measure the causal impacts of their smart city work on city outcomes, and how to prove the value for cities and citizens. A synthesis of the Council authority recommendations suggests that the design of smart city evaluation should be appropriate to the project, programme or city level, and to the innovation development maturity and scale of city projects. Evaluation approaches should reflect strategic city objectives and be open to improvement and evolution (as recommended by EIP-SCC 2013). Evaluation frameworks should be flexible, relevant and adaptable to different city challenges and circumstances. Some city authorities also considered that evaluation should have a diagnostic utility, helping cities identify both gaps in their smart city development and emergent innovation opportunities. Rather than focusing on arbitrary or easily-measured indicators, the choice of measures should include quantitative and qualitative, meaningful and comprehensive indicators that reflect the multi-faceted nature of smart cities and the complexity of urban systems. Overall, evaluation design should build on city data intelligence to support development of future city visions and strategies, which some authorities noted should be based more on a vision of liveable cities with embedded smart technologies rather than simply a digital city vision.

Conclusions

With the proliferation of smart city developments aiming to transform the urban context, it is important to identify and develop suitable methodologies to evidence the value, outcomes and impacts of smart city projects and programmes in complex urban contexts. The cities examined in the SmartDframe research were mainly project-focused in their evaluation work, and were at the early stages with their plans to evaluate the city-level impacts of smart city developments. This paper therefore aims to inform urban discourse on the development of appropriate, valid, credible and useful approaches to smart city evaluation by reviewing conceptual, measurement and evaluation issues, and examining the SmartDframe case-study research findings on city evaluation approaches to smart city development. This aims to guide evaluation practice applicable to smart city developments and support further research in this area.

The review of smart city models, measurement frameworks and indexes, identifies tools available to support high-level urban planning, development and benchmarking activities, together with a wide range of measurable indicators available to support evaluation of smart urban developments. Much work on smart city measurement has been directed at the city-level, while the SmartDframe study showed that the cities’ evaluation work has been primarily focused on micro-scale innovation projects, where evaluation is needed. This supported a major European study’s findings that most smart city solutions were niche, pilot innovation projects, which were at early stages of implementation and developed at limited geographical scales rather than developed explicitly city-wide (EU Directorate-General 2014). This raises challenges: of the applicability of smart city-level indicators and metrics to measuring the outcomes of smart city projects and programmes; and of the contribution of project measured success outcomes to the measurement of city-level performance. The CITYkeys initiative has begun to identify corresponding project and city indicators of smart city success outcomes, although recent findings revealed that most project-level success indicators could not be linked quantitatively with corresponding city-level indicators (Bosch et al. 2017). This suggests that the selection of measurement indicators for evaluation purposes should be appropriate to smart city developments at project, programme and city levels, and different geographical scales, with potential correspondences mapped between each level.

A key challenge for evaluation design is in developing standardized smart city development and performance indicators that provide meaningful, city and citizen-centred evaluation. Currently, national and international standardization initiatives play an important role in developing smart city standards and metrics, shaping city approaches to urban development. Moreover, standardized measurement indicators offer value for development policy and the potential for transforming governance (Holman 2009). However, several city authorities in the SmartDframe study were unconvinced of the need for new standardized, specific-smart city KPIs and frameworks, unless they were sufficiently relevant and adaptable to their unique city and project circumstances. An alternative preferred by some authorities is to measure the impacts of smart city developments against existing city KPIs aligned with city strategies. This would serve to leverage the value of embedded smart technologies for urban policies and actions.

A further key evaluation challenge for cities is to determine the value and causal impacts of smart city projects and programmes. Assumptions of smart technological determinism of outcomes, and the appropriateness of attributing measured city outcomes to smart city developments, are problematic in complex urban contexts. Evolutionary-systems perspectives on complexity suggest that the impacts of programme interventions can be best determined by setting a narrow scope for evaluation with clearly specified spatial and temporal system boundaries to control extraneous influences (Arnold 2004). New urban data sources from smart city infrastructures, sensor networks, space platform technologies, web services and social media, together with data-driven analytics, can help address this challenge through informing complexity modelling and better understanding of cause-effect relationships in cities.

Emerging from the SmartDframe study are recommendations that evaluation design needs to test theories underlying smart city development programmes and the benefits planned for citizens (see Pawson and Tilley 2004). Evaluation design should be appropriate to smart city project, programme and city levels of intervention and different city scales of development, and designed in collaboration with key city stakeholders with agreed success factors to support credible evaluation. Consideration is needed of the selection of evaluation methods, urban measurement indicators and data sources capable of determining the impact of smart city projects on citizens’ lives, and measuring the tangible/intangible and direct/indirect consequences of smart city developments. A range of qualitative and quantitative evaluation methods are available to identify value for specific purposes and stakeholder audiences (Oliveira and Pinho 2010; Edler et al. 2012; HM Treasury 2013), potentially informed by triangulation methods to link multiple, multilevel, multiscale evaluations to offer holistic urban policy evaluations (Magro and Wilson 2013).

Integrative approaches to evaluation design are needed to engage expert and community city stakeholders with evaluation processes, address project, programme and city levels of intervention and scales of smart city development, while capitalizing on big data opportunities and new generations of data analytics to inform city evaluation and its contribution to urban development. Embedding evaluation practices in city performance management processes is essential to determine the value, outcomes and benefits of smart city developments for cities and citizens. In this way, we can discern how effectively contemporary urban challenges are being addressed through a smart vision for future cities.

Disclosure statement

No potential conflict of interest was reported by the authors.

Funding

The Smart Dframe project is supported by funding from The Open University (OU) Smart Cities Open Challenge Competition. This is linked to the OU-led MK:Smart programme, which received significant funding from the Higher Education Funding Council for England.

Acknowledgements

The authors would like to acknowledge the valuable contribution of Gerd Kortuem to the SmartDframe research design and Lorraine Hudson who supported the research investigations. In particular, the city authorities of Birmingham, Bristol, Manchester, Milton Keynes and Peterborough are acknowledged with gratitude for their valuable contribution to the research. Figures 2‒6 derive from the Oculus Rift Virtual Reality system’s Google Earth VR app. The authors acknowledge the use of the Visualization for Terrestrial Informatics facility at Cranfield University, supported by NERC, UK (NE/L012774/1). Figure 7 derives from the ESRI Inc. ‘Urban Observatory’.