Public Innovation Capacity: Developing and Testing a Self-Assessment Survey Instrument

ABSTRACT

Governments around the world are pressed to develop innovative solutions for the wicked problems they are facing. To develop these solutions, the public innovation capacity—defined as the capacity to develop and realize new ideas for societal problems—is of crucial importance. This article builds a model of the public innovation capacity on the basis of innovation systems theory and the literature on public innovation. Five functions—mobilizing, improvising, vitalizing, balancing, and coordinating—are identified and operationalized to construct a self-assessment survey instrument for governments. This instrument in tested by using it to self-assess the public innovation capacity of a Dutch municipality. This test highlights the validity of the instrument and shows that it helps to identify weaknesses in the public innovation capacity. This measurement can form the basis for a reflection on the government organization’s public innovation capacity as a basis for interventions to improve this capacity.

KEYWORDS:

• Public innovation
• Organizational capacity
• Self-Assessment

Introduction

Innovation in general means bringing new ideas into implementation (Moore & Hartley, 2008; Sørensen & Torfing, 2011) and public innovation specifically refers to new ideas for societal challenges (De Vries, Bekkers, & Tummers, 2016). Innovative approaches to societal problems are argued to be in dire need in a situation of fiscal stress and wicked problems (Lægreid, Roness, & Verhoest, 2011; Wynen, Verhoest, Ongaro, & Van Thiel, 2014). These approaches demand that governments strengthen their public innovation capacity defined as their capacity to generate and implement new ideas for public problems (Gieske, Van Buren, & Bekkers, 2016). To be able to take appropriate action for strengthening their public innovation capacity, organizations need an instrument to measure this capacity. This article presents this self-assessment instrument.

This article will develop a self-assessment instrument of the public innovation capacity on the basis of two bodies of literature: theories of public innovation (Bason, 2010; Bekkers, Edelenbos, & Steijn, 2011; Borins, 2014; Farazmand, 2004; Gieske et al., 2016; Glor, 2008; Osborne & Brown, 2005; Walker, 2008) and theories of innovation systems (Freeman, 1987, 1995; Geels, 2002; Hekkert, Suurs, Negro, Kuhlman, & Smits, 2007; Hoogma, Kemp, Schot, & Truffer, 2002; Nelson & Nelson, 2002). In the literature on public innovation, Gieske et al. (2016) have developed a multilevel model of public innovation capacity that highlights the importance of individuals, organizations, and networks in public innovation. In the literature on innovation systems (Bason, 2010; Freeman, 1995; Hekkert et al., 2007), functional models have been developed to highlight how governments can assume a system responsibility for public innovation. These two literatures are brought together to identify the key functions of the public innovation capacity that will form the basis for the self-assessment.

In line with the literature on innovation systems (Freeman, 1995; Glor, 2008; Hekkert et al., 2007; Nelson & Nelson, 2002), the core assumption of the model is that all functions need to be fulfilled to make the public innovation system “work.” Bason (2010, p. 219) stresses that a self-assessment instrument can help to identify weak spots that needs to be addressed or potentials that should be realized. The self-assessment instrument operationalizes key functions of the innovation system to measure perceptions of stakeholders. These measurements identify strengths and weaknesses and form a basis for a reflection on the public innovation capacity. An empirical analysis of the Municipality of Utrecht in the Netherlands is used to show the value of the model for analyzing strengths and weaknesses in its public innovation capacity.

This article contributes to our academic understanding of public innovation by developing and testing a self-assessment instrument on the basis of a model of the public innovation capacity of government organizations. The perspective of public innovation systems helps to make a comprehensive analysis of the public innovation capacity that focuses on functions in a broader web of relations with a variety of actors rather than organizational tasks and provides governments with reflections that may help to strengthen their approaches to building a public innovation capacity.

Building a model of public innovation capacity

While the study of innovation in the private sector has a long history, attention for public innovation only started in the 1980s (Bekkers et al., 2011; Berry & Berry, 1990; Borins, 2000; Brown, Berry, & Goel, 1991; Osborne & Brown, 2005; Walker, 2008). A key theme in the study of public innovation is the tension between the dynamics of entrepreneurial activity on the one hand and the stability—or inertia—of democratic and administrative institutions on the other (Van Duivenboden & Thaens, 2008). Borins (2000), however, emphasizes that public innovators combine initiative and creativity with respect for the law and due process. This observation is now widely accepted, and innovation in government is largely regarded as a desirable and important way to enhance the production of public value.

To make public innovation work, governments require a public innovation capacity. The idea that governments need a new type of capacity that enables them to tackle the new world of globalization and technological change has been expressed by Farazmand (2009). He stresses that this capacity needs to entails the capacity to act more adaptively in able to tackle unforeseen problem situations. To develop the idea of this capacity more specifically for public innovation, we build upon Moore and Hartley (2008) definition of innovation, who define the public innovation capacity as the capacity to develop and realize new ideas for societal problems. This capacity, as Gieske et al. (2016) indicate, does not only mean that creativity and experimenting are stimulated but it also entails the capacity to connect and facilitate collaboration, the capacity to combine exploitation and exploration and the capacity to absorb new knowledge and continue to learn. These different capacities are needed to enable the (continuing) realization of public innovations.

Empirical research into public innovation has expanded rapidly in the best decades (for overviews, see Bekkers, Tummers, & Voorberg, 2013; De Vries et al., 2016). While much work is based on case studies and qualitative research, currently more quantitative research into public innovation is being conducted (Arundel & Huber, 2013). Borins (2000) has conducted important analyses of entries to innovation awards to identify key features of public innovation. Many of the investigations focus on the dissemination of innovation (Berry & Berry, 1990; Lapsley & Wright, 2004; Walker, 2006) but interesting work is also being done in the measurement of different types of innovation (Walker, Jeanes, & Rowlands, 2002). Several authors have also focused on measuring barriers and drivers of public innovation (Bloch & Bugge, 2013) and others highlight the difficulties of establishing what an innovation is on the basis of answers to surveys (Arundel & Huber, 2013). The instruments measure various aspects of public innovation processes but do not provide information about the public innovation capacity.

A multilevel framework for the assessment of the public innovation capacity has been developed by Gieske et al. (2016). They highlight that the public innovation capacity needs to be assessed at the level of the individual, the organization and the network and needs to take into account the connective, ambidextrous, and learning capacities. This results in a strong framework for analysis that summarizes much of the literature on the topic. At the same time, they did not translate this framework into an instrument for assessing the public innovation capacity. A complicating factor is that their nuanced perspective focuses on so many issues that it is difficult to translate this into a self-assessment instrument. For that reason, this article will build upon the work by Gieske et al. (2016) but takes a different approach and focuses on functions in innovation systems rather than on actors in different levels. This approach follows the suggestions made by Glor (2008) to study innovation from the perspective of complex, adaptive systems. This article aims to contribute to our understanding of public innovation, by bringing in a theoretical perspective that helps to position public and collaborative innovation within the public innovation system.

The perspective of the innovation system has been developed by historians, to account for different approaches to and trajectories of innovation in different countries (Freeman, 1995). A key feature of this approach is that it conceptualizes innovation as both an individual and a collective act. Freeman (1987) defines an innovation system as the network of institutions in the public and private sectors whose activities and interactions initiate, import, modify, and diffuse new technologies. This system approach challenges the assumption that differences between countries should be understood on the basis of quantitative R&D comparisons and proposes that institutional differences are crucial to understand these differences.

Earlier studies of innovation systems—such as Freeman’s (1995) seminal work—emphasize structural characteristics of these systems. These approaches, however, suffer from “institutional determinism” in the sense that there is little attention for actual activities of actors in the system and policy choices (Hekkert et al., 2007, p. 414). For this reason, this article follows Hekkert et al. (2007), by putting emphasis on processes or functions in innovation systems, rather than on the structural features of these systems. According to Hekkert et al. (2007, p. 414), these functions are highly important to the performance of innovation systems: “If we knew what kind of activities foster or hamper innovation—thus, how innovation systems ‘function’—we would be able to intentionally shape innovation processes.” This assortment is in line with Glor’s (2008) observation that we need to understand the organizational patterns in complex innovation systems to understand how innovation can be strengthened. Hekkert et al. (2007) highlight that these activities can be labeled as functions when they contribute to the goal of innovation systems which is to generate and diffuse innovations (see also Jacobsson & Bergek, 2004; Johnson, 2001).

Hekkert et al. (2007, p. 414) stress that an innovation system is “a heuristic attempt, developed to analyze all societal subsystems, actors, and institutions contributing in one way or the other, directly or indirectly, intentionally or not, to the emergence or production of innovation.” The system is not “out there,” but constructed in actions and communications by scholars to enhance our understanding of the dynamic interactions between various actors in an institutional context. Checkland (1981) has developed a Soft Systems Methodology that highlights the system is being created in purposeful actions and setting the boundaries of the system is a key decision in the attempt to develop a useful heuristic. We will use the innovation system approach in this article to conceptualize the Public Innovation System and to identify the different functions of public innovation.

In the literature, various functions of innovation systems have been identified (for an overview, see Hekkert et al., 2007). These functions are generally developed for private-sector innovation and are therefore not directly suitable for studying public innovation systems. An exception is Bason (2010) who identifies four functions—consciousness, capacity, co-creation, and courage—for public section innovation. We build upon this overview of functions, but connect it better to more established theories of public innovation. A generic perspective on the evolution of innovations forms our starting point. McKelvey (1997) uses the main principles of evolutionary economics—variety, selection, and retention—to develop a general overview of three functions: generation of novelty leading to diversity, selection among alternatives, and retention and transmission of information. Translated to the public sector, these functions can be labeled as mobilizing, experimenting, and institutionalizing. Two other functions need to be added to cover the normative issues involved in innovation: balancing, and the coordination between the different functions in the public innovation system (coordinating).

These components form the basis for a model of the public innovation capacity. This model stresses that governments need to fulfill five functions—mobilizing, experimenting, institutionalizing, balancing, and coordinating—to stimulate the public innovation system. It is important to realize that the different functions are not sequential, since they can interact and generate feedback loops (Hekkert et al., 2007, pp. 425–427). Successful test situations (experimenting) or demonstrated uptake in work routines (institutionalizing) may, for example, result in easier mobilization of new stakeholders. Or lack of sound coordination may result in a disconnect between experimenting and institutionalizing and, consequently, a decline in engagement of stakeholders (mobilizing). In that sense, the model should be seen as a dynamic model that does not only provide insights in current performance, but also in the changes over time.

The first function, mobilization, entails connecting to other actors through boundary spanning activities (Gieske et al., 2016). Stimulating other actors to engage in innovation processes is the starting point for processes of collaborative innovation. Public innovation is increasingly perceived as collaborative innovation. Top–down approaches based upon strong corporate involvement have been rejected and more participatory bottom–up approaches have been proposed (Townsend, 2014). These approaches build upon notions such as collaborative innovation (Gloor, 2006; Sørensen & Torfing, 2011) and user innovation (Baldwin & Von Hippel, 2011; Von Hippel, 2005). Dente, Bobbio, and Spada (2005) stress that the mobilization of a variety of actors contributes to the innovation potential, since more complex innovation networks result in a better performance of the public innovation system. This means that mobilizing the ideas, knowledge, (financial) support, and operational capacity of other actors is a crucial requirement for successful public innovation systems.

Bason (2010, p. 118) highlights that organizations must leverage the innovation capacity of all civil servants, and not just dedicated project teams or R&D departments. Large and small companies, social entrepreneurs, societal organizations, and knowledge institutes are often mentioned as important actors in public innovation processes (Gloor, 2006; Sørensen & Torfing, 2011). A specific group to be mobilized for innovation is citizens who can help to assess what is valuable and engage in the innovation processes as end users (Baldwin & Von Hippel, 2011; Von Hippel, 2005). A final group of actors to be mobilized is other governments. These may be fellow local governments that share their innovations, but also central governments who facilitate and even support innovation processes.

The second function is experimenting. Innovation requires the flexibility to act against or in the absence of formal procedures (Blau, 1956). Mulgan (2009) highlights anything really innovative is almost certain not going to go according to plan. This means a different approach is needed than normally applied in bureaucratic organizations: experimenting instead of planned implementation. In the literature, next to experimenting, words such as co-creation (Voorberg, Bekkers, & Tummers, 2015) and design thinking (Bason, 2010) are used to describe this different dynamic in government bureaucracies. We choose to use the word “experimenting” for this function, since it puts the emphasis on how this activity is different from regular activities in bureaucratic organizations (see also Orlikowski, 1996). For governments, the notion of experimenting completely contradicts the Weberian bureaucratic logic of predictability and standardization.

In the literature on innovation, the idea of an “experimental space” is well established. The literature on strategic niche management emphasizes the need to develop innovation in friendly “incubation rooms” or “niches,” before they are implemented in the “real world” (Hoogma et al., 2002). Creating robust and fertile conditions that allow for experimentation is crucial to success. Karvonen and Van Heur (2014) highlight that the concept of experimentation builds upon attractive notions, such as creativity and practice to translate distant policy targets into concrete action domains. There is a growing body of literature on experimentation in the public domain (Evans, Karvonen, & Raven, 2016). This literature acknowledges the important of citizen-centric living labs to facilitate social learning. Sengers, Berkhout, Wieczorek, and Raven (2016) indicate that experiments involve a broad coalition of actors that engage in social learning related to a socio-technical configuration, to generate both proprietary and public knowledge. The capacity of the innovation system to make these experiments “work” is crucial to their success, and this means obtaining financial means and securing political, administrative and societal support.

The third function is institutionalizing. A key finding in research into successful public innovations is that they need to be connected to the broader institutional environment (De Vries et al., 2016; Osborne & Brown, 2005). Institutionalizing entails a set of activities that start with evaluating experiments and “packaging” the innovation for dissemination, upscaling, embedding, and routinizing (Meijer, 2014). Securing long-term collaboration with partners is also a key challenge for this function (Sørensen & Torfing, 2011). While the niches or incubation rooms create opportunities for developing and testing new ideas, their large-scale value can only be established if these experiments are translated into organizational practices. Innovation is in the end about embedding new ideas in existing structures (Geels, 2002; Hoogma et al., 2002). Gieske et al. (2016) refer to this as balancing exploitation and exploration.

The idea that the experiment should have a broader impact has been widely acknowledged, and notions such as upscaling, replicating, transforming, seeding, rolling out and breaking through are used to refer to this broader impact (Evans et al., 2016). Upscaling is the most frequently used term and Cooley and Kohl (2005) identify three forms: expansion as upscaling within the organization that developed it, replication as upscaling by other organizations and spontaneous diffusion as the spread of good ideas on their own account. These notions suggest that an invisible hand guides these resulting changes and the actors and processes through which the broader impact is generated are often not acknowledged or analyzed. Bason (2010, p. 200), however, stresses that scaling is not a mechanistic or easy process, but only occurs when the target group is open to the idea of adopting innovations that have been developed elsewhere. Generating broader impact does not mean that the innovations are simply copied: adaptation, tinkering with the innovation, is a normal aspect and often crucial to make the innovation fit different contexts (Brandsen, Cattacin, Evers, & Zimmer, 2016; Hoogma et al., 2002).

The fourth function is balancing, which means deliberately weighing the different interests and values involved in the process of public innovation and choosing the option that contributes most to societal desirable outcomes (Hull, 2000; Macaulay & Norris, 2013). Balancing is different from the other functions, in the sense that balancing does not seem to be a necessary condition for advancing the innovation process. Balancing might even halt or slow down the processes by bringing in extra considerations and difficult issues. At the same time, balancing is crucial to realizing responsible innovation in the sense that the innovation does not harm stakeholders or neglect key interests and to ensure desirable and acceptable research outcomes (Stahl, 2013).

A crucial aspect here is the politics of experimentation: who is allowed to participate, whose ideas are developed, who decides about institutionalization (Evans et al., 2016). The literature on responsible innovation emphasizes the need for a fair process with input from relevant stakeholders. The notion of (participatory) technology assessment is helpful for developing an understanding of this function (Rip, Misa, & Schot, 1995). Balancing entails identifying risks, disadvantages and tensions and discussing these to make deliberate choices. A public debate about sensitive issues is important, and ethical leadership is needed to generate fruitful interactions and support for choices. Finally, the capacity to mediate conflict is important to deal with the tensions that may emerge around public innovations.

The final function—coordinating—emphasizes the need to manage the innovation system (Dente et al., 2005). Coordinating is about managing the connections, interfaces and relations between the various functions in the innovation system (Hekkert et al., 2007). Important are, for example, the connections between experimenting and institutionalizing. Coordinating the innovation system also entails the provision of certain generic functions. For one thing, not only creatives and “tinkerers” are needed (Glor, 1998), but also sponsors and champions are crucial to innovation (Meijer, 2014). Crucial are also individuals that are capable of identifying the potential of new technologies. Coordinating means that the various actors who play a role—creatives, tinkerers, sponsors, and champions—are connected to a promising innovation, and to other actors in the public innovation system.

At a more general level, innovation leadership is crucial to stimulating a climate of innovation (Bossink, 2007). A culture of innovation needs to be nurtured, to provide support for civil servants, citizens, and other actors who take risks in their attempt to develop new and promising approaches to wicked problems (Lægreid et al., 2011). Innovation also needs to be supported by building innovation capacity into the system, through training and making resources available. Presenting the activities in a convincing storyline is crucial (Bason, 2010, p. 175). A language of innovation is important to mobilize the variety of actors. This framing is also important for obtaining support from the institutional environment—legal, political, administrative, societal—for the innovations. Bason (2010, p. 254) stresses that an organization—or an innovation system—needs to have a “license to innovate”: the (institutional) environment needs to be supportive to innovation.

The model of the Public Innovation Capacity that we have developed consists of five functions: mobilizing (M), experimenting (E), institutionalizing (I), balancing (B), and coordinating (C). The assumption of innovation systems theory is that these functions need to be fulfilled to make the system “work” (Hekkert et al., 2007). The relation can be presented in the following equation:

$\mathit{P}\mathit{I}\mathit{C}=\mathit{m}\times \mathit{e}\times \mathit{i}+\mathit{b}+\mathit{c}$

The equation highlights that the output in terms of innovations that are embedded in the organization is the result of mobilizing, experimenting, and institutionalizing. Balancing is added since this does not influence the process of variation, selection, and retention but it is important for the legitimacy of the innovations. Coordinating is added since the permanent capacity to innovate depends on managing the internal and external interfaces.

To measure their public innovation capacity and to identify strengths and weaknesses, organizations need to assess their performance on these five functions. The next section will indicate how we have developed an instrument for the self-assessment of the public innovation capacity that builds upon this model.

Method

Data innovation as a testbed for instrument development

To develop a reliable and valid self-assessment instrument for the public innovation capacity, we use data innovation in the Municipality of Utrecht as a specific domain. Data innovation is a form of innovation that is of crucial importance to cities around the world and that is attracting much attention (Chen & Hsieh, 2014; Mergel, Rethemeyer, & Isett, 2016). We use this form of public innovation for developing a self-assessment instrument for three reasons. Firstly, this form of innovation requires collaboration between various actors within and outside the organization and thus stresses the need to mobilize various stakeholders. Secondly, this form of innovation requires specific types of expertise and knowledge that are available inside but also outside the organization. Thirdly, this type of collaboration requires the connection between people with knowledge of technology and people with knowledge about the organization and its environment. This means that this domain is highly suitable for studying socio-technical innovation but the findings cannot directly be generalized to social innovation. We will come back to this issue in the discussion.

Item generation on the basis the literature

To develop a reliable and valid instrument, well-known quality controls for the development of a measurement instrument, as described by DeVellis (2003), were applied (for similar approaches, see Netemeyer, Bearden, & Sharma, 2003; Crook, Shook, Morris, & Madden, 2010; for other applications of DeVellis’ method, see Govindarajan & Kopalle, 2006; Tummers, 2012). We determined what we wanted to measure on the basis of the literature on (public) innovation, generated an item pool and determined the format for measurement. The five functions of the public innovation capacity were operationalized on the basis of the literature to develop an instrument for measuring this capacity in public organizations. For each function, four or five statements were developed and the respondents were asked to indicate on a 5-point Likert scale whether they agreed or disagreed with the statement that the Municipality of Utrecht was doing well on the various functions. The statements were used to develop scales for each function (DeVellis, 2003).

Qualitative refining and expert review of the instrument

The initial instrument was applied through interviews to test its relevance and completeness. These interviews were used to identify key aspects of the functions in the context of data innovation in local government and refine the items. Interviews were conducted with 11 respondents with different relations with the Municipality of Utrecht and the domain of data innovation to refine the instrument that we had developed on the basis of the literature. The respondents were asked to score the municipality’s public innovation capacity on items for each of the five functions. The idea of the method is to conduct a 360° assessment (Toegel & Conger, 2003) of the public innovation capacity by interviewing insiders, those responsible within the organization, the internal target group of civil servants, the external contact and the democratic representatives. The first group consisted of civil servants with direct knowledge of these innovation processes (R1, R7, R10), the second group of those hierarchical responsible for data innovation (R8, R6), the third group of civil servants at more distance from these processes (R2, R3), the fourth group consisted of knowledgeable outsiders (R4, R11), and the fifth group of council representatives (R5, R9). In the interviews, respondents were asked whether they thought all relevant items were measured. Some items were added on the basis of the interviews (e.g., evaluating experiments). In addition, we presented the instrument at an international public management conference to solicit feedback from experts. The comments from the experts focused on connecting the model better to the broader literature and highlighting the role of institutional context. The interviews and expert comments were used to adapt the scales and add items. The instrument is presented in table 1 below.

Table 1. Instrument for measuring the public innovation capacity

Function	Statements for self-assessment
Mobilizing	M1. Employees in City X with ideas about data-driven innovation easily find the right persons in the city to jointly realize these ideas.*
	M2. The people in charge of data-driven innovation in City X succeed in engaging companies, researchers and citizens in the development of new ideas.
	M3. City X has a strong structural network of companies, researchers and citizens connected to data-driven innovation.
	M4. The people in charge of data-driven innovation in City X succeed in stimulating the development of new ideas among colleagues in City X.
	M5. City X has a strong network of employees with an interest in data-driven innovation.
	M6. A company, researcher or citizen with good ideas for data-driven innovation easily finds the right persons within City X to develop these ideas further.
Experimenting	I1. City X is successful in setting up experiments.
	I2. City X has societal support (from citizens, NGOs, companies, etc.) for experiments on data-driven innovation.**
	I3. Political institutions in City X – representatives, aldermen – support experiments with data-driven innovation.
	I4. The administrative executives of City X support experiments with data-driven innovation.
	I5. City X makes sufficient funds available for experimenting.
	I6. If necessary, City X engages other governments, companies and societal organizations in experiments around data-driven innovation.
Institutionalizing	R1. City X is successful in scaling up experiments.
	R2. City X adopts data-driven innovation that have proven to be successful on a small scale in the organizational routines.
	R3. City X evaluates experiments with data-driven innovation well.
	R4. City X succeeds in turning experimental collaboration with governments, companies and societal organizations into structural forms of collaboration.
Balancing	B1. City X succeeds in identifying risks, disadvantages and tensions around data-driven innovation.
	B2. City X initiates the public debate about the risks, disadvantages and tensions around data-driven innovation and how to deal with these.
	B3. If there are conflicts, City X is good at mediating conflicts around data-driven innovation.
	B4. In City X, ethical aspects of data-driven innovation are discussed well.
Coordinating	C1. City X makes financial means available for data-driven innovation on a structural basis.
	C2. There is a good exchange of information on data-driven innovation between all actors in City X.
	C3. City X has a culture that stimulates data-driven innovation.
	C4. City X creates the right conditions for data-driven innovation (training, information exchange, instruments, etc.).
	C5. City X has a clear vision on data-driven innovation.
	C6. Political institutions in City X – representatives, aldermen – are prepared to allocate financial means in the budget for data-driven innovation.

* ‘City X’ and ‘data-driven innovation’ are the templates in this instrument (DeVellis, 2003). They can be adapted to specify a city and a domain of innovation. The questionnaire could also be applied to measure for example social innovation, ICT-innovation or environmental innovation.

** I2 was removed to construct a reliable scale.

Testing the self-assessment instrument

The value of the self-assessment instrument is explored by testing it on a case. The case selected for this test is data innovation in the Municipality of Utrecht. Utrecht is a mid-size city in the Netherlands and home of several research institutes and a university. The city has a large population of highly educated people and contains many ICT-companies and the electorate leans to the left. The current government coalition has identified data innovation as one of the key means to prepare the municipality for the future and the City Council has agreed to make substantial financial resources available for this form of innovation. The Municipality of Utrecht uses data to improve (real-time) monitoring, predictive analysis, profiling, and data discovery. These new capabilities are used to drive improvements in policy areas as diverse as security, welfare policies, public health and waste management. The new approach does not only entail the use of data through technological systems; it also means that the municipality tackles policy problems in a different manner. Utrecht has created a specific function—the coordinator of data innovation—to stimulate data innovation. The budget is used for training of civil servants, new ICT facilities, management of (open) data, stimulating pilot studies and external advice. This budget and these activities qualify the Municipality of Utrecht as one of the frontrunners in data innovation. This case is analyzed with the instrument we developed to identify strengths and weaknesses in the public innovation capacity.

For the self-assessment, the questionnaire was sent to everybody who had attended a meeting on data-driven innovation in the Municipality of Utrecht or had been working on any type of project: 310 respondents (220 working for the municipality and 111 external contacts). We received 104 valid responses (34%). 68.9% of the respondents works for the Municipality of Utrecht, 28.2% for another organization and 2.9% for both (e.g., through a temporary position). The respondents were on average 44,5 years old and 71% men and 28% women (and 1% other). The respondents were selected on the basis of their knowledge of these innovation processes and we found that 66% indicated that they knew (very) much about this issue. We checked for correlation with control variables such as age, gender, level of education, interest in this type of innovation, knowledge about the innovation and number of months engaged in this type of innovation. The only control variable that had a slight impact is whether they worked for the Municipality of Utrecht or not. Externals were slightly more positive about the Municipality of Utrecht’s innovation capacity (p < 0.01). In the analysis, the scales that were developed for the different dimensions were tested. If the scales had a Cronbach’s α of <0.7, we tried to see if the reliability could be strengthened by leaving out certain items. For each function, we calculated the average score and the standard deviation.

Finally, the results of the self-assessment were published in a report for the Council of the Municipality of Utrecht and the findings were presented in a meeting on March 24, 2017 to the Executive Board of the Municipality to test the value of the instrument. The Executive Board was asked whether they recognized the results and whether they formed a basis for interventions aimed at strengthening the municipality’s public innovation capacity.

Testing the instrument: public innovation capacity of the municipality of Utrecht

The first function of the public innovation capacity of the Municipality of Utrecht, mobilizing, was measured by asking the respondents to assess six questions related to the mobilization of employees, companies, scholars and citizens. We assessed the reliability for the scale we had developed to assess this function and found a Cronbach’s α of 0.792 which, according to Kline (1999), makes for a reliable scale. The score for mobilizing in the quantitative research was 3.46 (SD = 0.703) on a five-point scale which make it the second highest score of the five functions.

The second function, experimenting, was assessed by asking the respondents six questions about setting up experiments and obtaining funding and support from administrative executives, political institutions and society. The Cronbach’s α for this function was 0.868, but only after the question about societal support (I2) was removed from this scale. According to Kline (1999), this can certainly be regarded as a reliable scale since it is well above the cut-off point of 0.7. The score in the quantitative assessment for experimenting was the highest score of the five functions: 3.83 (SD = 0.692).

The third function, institutionalizing, was measured by asking four questions about evaluating experiments, scaling up experiments, adopting innovation into routines and turning incidental collaboration into structural collaboration. The Cronbach’s α for this function was 0,717 which, according to Kline (1999), is reliable. The score in the quantitative assessment for institutionalizing was the lowest score of all functions: 2.97 (SD = 0.750).

The fourth function, balancing, was measured by asking four questions about identifying risks, initiating public debates, mediating conflicts and discussing ethical aspects. The Cronbach’s α for this function was 0,696 which is just below 0.7, but according to Kline (1999), can still be regarded as reliable for constructs with a level of diversity. The score in the quantitative assessment for experimenting was 3.12 (SD = 0.791) which make it the second-lowest scoring function.

The fifth function, coordinating, was measured by asking six questions about financial means, exchange of information, an innovation culture, condition for innovation, a vision and budgetary support from political institutions. The Cronbach’s α for this function was 0,827 which, according to Kline (1999), is certainly reliable. The score in the quantitative assessment for experimenting was 3.32 (SD = 0,762) which make it the middle score of the five functions.

An overview of the average scores for the five functions is presented in Table 2.

Table 2. Average scores for the five functions.

The table provides some interesting empirical findings about the public innovation capacity of the Municipality of Utrecht. The findings show that the respondents think that the Municipality is doing fairly well in the mobilizing, experimenting and coordinating innovation functions. Well-known barriers to innovation such as siloization, risk avoidance, focus on production processes, etc., are mentioned, but the Municipality of Utrecht seems to have found ways to deal with these barriers since it scores fairly high on mobilizing and experimenting. However, it falls short in institutionalizing successful experiments, as well as balancing the different values in the innovation process. Key challenges are embedding innovations in processes and starting a debate about ethical implications of innovation. These shortcomings can possibly be attributed to relative recent efforts to strengthen the innovation capacity. Efforts were targeted at mobilizing and experimenting to change the innovation climate in the municipality. Now that the capacities to mobilize internal and external stakeholders and set up experiments have been built into the organization, a next step is to use the lessons from the experiments to improve the whole organization and to balance the various values at stake.

On presentation of these outcomes to the Executive Board of the Municipality of Utrecht they stressed that they recognized the results. They acknowledged the focus on mobilizing and experimenting and indicated that there was a need for a stronger focus on institutionalizing successful experiments. The municipality has now set up a platform for drawing lessons from experiments and embedding these in the organization and they have been developing new instruments for balancing values in innovation such as a Data Ethics Decision Aid.

Discussion

The use of the self-assessment instrument for measuring the public innovation capacity has indicated that the instrument resulted in valuable insights for local government. The findings resonated with current debates and were used to take new actions for improving the municipality’s public innovation capacity. This indicates that the self-assessment instrument that was developed on the basis of the literature provides value for an organization. At the same time, the test of the instrument has important limitations.

A first limitation of the test is that it presents a self-assessment of the public innovation capacity. Most respondents were employees of the Municipality of Utrecht. We found that external respondents were slightly more positive about the municipality’s public innovation capacity but these external respondents were not neutral outsiders but worked for universities and consultants who collaborated with the municipality is this field. In that sense, this measurement provides an internal perspective on the public innovation capacity that can stimulate debate and internal learning. Theoretically, the instrument could also be used for an external assessment but that would require external outsiders with sufficient knowledge about the organization to answer these questions. This also suggests that it is different to compare different domains of innovation with one organization since this requires respondents with knowledge about these different domains (see also limitation 3 below).

A second limitation is that the test only measured the public innovation capacity at one moment in time and measuring at different moments is crucial to understand its dynamics. The model assumes that a government organization is capable of realizing all these functions at the same time but the academic literature highlights that there may be a tension between the functions of experimenting and institutionalizing (Gieske et al., 2016) and between experimenting and balancing (Stahl, 2013). The measurement in the Municipality of Utrecht highlights the increased emphasis on mobilizing and experimenting and identifies institutionalizing and balancing as falling short. The model suggests that all these functions need to be fulfilled but the theory suggests that actions to enhance the functions of institutionalizing and balancing may have a negative effect on mobilizing and experimenting. Longitudinal research is needed to assess whether the functions can be applied in a synergistic manner or whether we will find trade-offs between the functions.

A third limitation of the empirical test of the self-assessment instrument is that it was tested on a specific domain of public innovation: data innovation. This type of innovation is rather specific in its technical nature which may result in other dynamics than other field of innovation (Bekkers et al., 2011; Brandsen et al., 2016). Experimenting, for example, may be more important in this domain than in others and institutionalizing may be more difficult in other domains. The qualitative interviews clearly indicated that the respondents needed to answer these questions specifically for data innovation since they had no knowledge about other domains of innovation. The resulting score provides input for debates about innovation in this domain but not for others. This highlights that measuring the public innovation capacity of an organization according to this approach provides information about a specific domain but not about the organization in general.

A final limitation of the test is that we did not test whether the model is complete and whether all relevant functions have been identified. The functions were identified on the basis of a theoretical argument and tested through qualitative interviews. In addition, the presentation of the results to the Executive Board of the Municipality of Utrecht did not result in the identification of other functions. At the same time, we have not made an effort to systematically identify other functions. More inductive research could help to check whether there are other functions that are crucial to the public innovation capacity of government organizations.

Conclusions

This article set out to develop a model and a measurement instrument for the Public Innovation Capacity of government organizations. We used theories on public innovation (Bekkers et al., 2011; Borins, 2014; Farazmand, 2004; Osborne & Brown, 2005; Walker, 2008) and theories of innovation systems (Freeman, 1995; Geels, 2002; Hekkert et al., 2007; Hoogma et al., 2002) to develop this model. In our model of the public innovation capacity, these functions have been labeled as mobilizing, experimenting, institutionalizing, balancing, and coordinating. On the basis of the model, this article developed an instrument for measuring the public innovation capacity. This instrument was applied in the Municipality of Utrecht to measure the public innovation capacity and to test the self-assessment instrument.

The basis for the self-assessment instrument is our model of the public innovation capacity. This model integrates the literature on innovation ecosystems and public innovation and presents a functional perspective on the public innovation capacity. The functional perspective helps to analyze why governments are or are not successful in stimulating public innovation in their cities by focusing on their contributions to the system. More specific contributions include the identification of specific activities such as managing the institutional environment, managing internal connections, and managing the technological infrastructure as important activities for the public innovation system.

The key contribution of this article to the academic literature is the self-assessment instrument for measuring the public innovation capacity of a government organization. The article shows how the self-assessment instrument can map perceptions of government performance on the various functions. The instrument provides specific insights into strength and weaknesses, but also an overall assessment shared by a large group of respondents. These measurements can form the basis for a reflection on the government organization’s public innovation capacity as a basis for interventions to improve this capacity. The value of this model is explored through a case study in the Municipality of Utrecht in the Netherlands. The explorative research shows that the Municipality of Utrecht has started experiments and coordinated different innovation efforts, but its capacity to mobilize innovative forces, to capitalize on the experiments and to weigh different values and interests could be developed further.

Further research could explore the value of this instrument in two directions. The first direction is a focus on the development of the system over time. The self-assessment instrument now measures perception at one moment, but a richer understanding can be achieved by measuring how the public innovation capacity changes to understand the value of interventions in this capacity. The second direction is enhancing our understanding of the relation with the specific domain of innovation. The current analysis is based on an analysis of a rather technical domain of innovation. The theoretical model pretends to be broader but further research needs to show whether the instrument also works for the capacity to generate social innovations.