There’s a lid for every pot! The relationship between performance measurement and administrative activities in Italian ministries

Abstract

Following the spread of performance measurement (PM) in public organisations, a vast literature has developed on its functioning and difficulties. This paper intends to investigate the relationship between administration activities, which can by their nature have different characteristics, and PM systems. The application of PM in the Italian ministries through the analysis of 1,511 objectives/indicators shows the relevance of administrative activities in influencing the ability to measure performance and suggests that neglecting this link is detrimental to the evaluation system and undermines its ability to provide a reliable representation of the contribution of public organisation in relation to performance.

Keywords:

• performance measurement
• administrative activities
• public sector
• ministries

Public Interest Statement

In order to operate properly, performance measurement systems require awareness of their impact on organisations (reactivity), besides their function as technical and neutral tools. The use of performance measurement systems should be adapted to the characteristics of the administrative activities that these systems measure. With complex and ambiguous administrative activities, direct connection of the performance measurement system with performance-related pay mechanisms may generate resistance and favour gaming. With this kind of activities, it is preferable to use the indicators to explore the sense-making process of performance measurement in order to identify areas of actions that require particular attention and to highlight priorities.

1. Introduction

New public management (NPM) has generated a new momentum for the performance measurement (PM) systems, which were at the top of agenda in many countries, thus representing the pivot or one of the main elements in public administration reform programmes. Ingraham (2005) states that for much of the twentieth century—and certainly since the 1980s performance has been a siren’s song for nations around the world, mainly because although performance promises may be difficult to keep, they are the right promises as they are the core of every governance and accountability process in public administration.

Despite this wide dissemination, PM is often reported as on the weak points of NPM-based reforms. Instead of supporting NPM-inspired measures, PM can become an obstacle in reforms and a mere bureaucratic activity tending to self-legitimation rather than usefulness (De Bruijn, 2002), thus “hitting the target and missing the point” (Bevan & Hood, 2006, p. 521).

Like the NPM (Pollitt, 2009), PM has developed more on an operational level linked to its dissemination and operational implementation, often leaving out an in-depth theoretical reflection. On the other hand, the diffusion of PM took place on the basis of external guidelines and inputs rather than on the basis of internal requests and conscious organisational needs (Arnaboldi, Lapsley, & Steccolini, 2015; Dahler-Larsen, 2013; Diefenbach, 2009; Marra, 2017).

An analytical aspect to explore the topic of the measurement of administrative activities (i.e. activities produced by public administrations), which is less contingent yet structural and transversal, is the link between PM and the characteristics of the activity measured. Since Ouchi (1979), several studies have examined the measurability of the activities of public organisations (Abma & Noordegraaf, 2003; Davis & Stazyk Edmund, 2015; Ditillo, Liguori, Sicilia, & Steccolini, 2015; Frey, Homberg, & Osterloh, 2013; Mascarenhas, 1996; Noordegraaf & Abma, 2003; Speklé & Verbeeten, 2014; Vakkuri, 2010). This study sets out to contribute to this debate by examining the data obtained following the application of PM in the Italian ministries.

Overcoming the anecdotal nature of many analyses on the status of PM especially in non English-speaking countries (Nasi & Steccolini, 2008), this study, through a documentary review in 11 ministries, analyses 1,511 objectives and indicators used to measure the performance of administrative activities.

On the basis of the definition of ambiguity of the administrative activities proposed by Abma and Noordegraaf (2003), based on the level of routine of the administrative action and on the importance of the interactions with subjects that are external to the production process, this study has two objectives: first, to check whether, on the basis of the data collected, the type of activity measured is relevant in determining the ability of the measurement system to frame performance; second, on the basis of the empirical analysis, to formulate some considerations that are useful to raise awareness of the implementation of PM in the activities of the public administrations. In other words, this article intends to explore the connection between the characteristics of administration and PM in Italian ministries in order to draw useful considerations in relation to the use of PM systems in public administrations.

2. Literature review

Since the 1980s, the centrality of PM in the NPM paradigm is supported by the wide and transversal diffusion of the mechanisms and practices of PM in various nations and public sectors (Ingraham, 2005; Van Dooren, Bouckaert, & Halligan, 2010). This spread is far from having exhausted its propulsive thrust. In the face of a marked spread, the literature has analysed, without definitive results so far, the real contribution of measurement systems to the performance of public administrations (Poister, Pasha, & Hamilton Edwards, 2013) and more generally the difficulties encountered by PM systems. PM has been questioned by many and accused of betraying the promises associated with it (Perry, Engbers, & Jun, 2009; Sofyani, Akbar, & Ferrer, 2018; Ter Bogt, Van Helden, & Van Der Kolk, 2015; Van der Kolk & Kaufmann, 2018).

A critical approach towards the implementation of measurement systems that has found wide support is the approach aimed at finding significant and widespread weaknesses in the definition of objectives, indicators and in the link between indicators and objectives that appear weak and unable to represent the real contribution of public organisations in terms of performance (Cavalluzzo & Ittner, 2004). As some studies argue (Dahler-Larsen, 2013; Diefenbach, 2009; Van Thiel & Leeuw, 2002), these problems result from the difficulty of objectives and indicators to fully and effectively represent administrative actions, thus generating a so-called performance paradox (Meyer & Gupta, 1994), whereby the indicators used in the measurement lead to an evaluation that is completely different from the actually achieved level of performance. In other words, indicators loose their ability to tell the difference between good and bad performances and, “as a result, the relationship between actual and reported performance declines” (Van Thiel & Leeuw, 2002, p. 271). There is therefore a divergence between organisational objectives and measurement system (Dahler-Larsen, 2013; Smith, 1995), in which the former are measured and only partially represented by the latter. Given the operational and structural complexity of administrative procedures, measurement systems tend to focus on the most easily quantifiable and short-term, and therefore measurable, aspects of administrative action (“tunnel vision” and “measure fixation” phenomena, see Smith, 1995), leaving aside qualitative, value-related and ethical aspects, which are however central to public activities (Stewart & Walsh, 1994). As (also empirically) highlighted, indicators are more likely to measure more easily quantifiable aspects than more relevant aspects, thus creating an information gap (Bohte & Meier, 2000).

While some authors associate this problem with the intrinsic complexity of public action, others point out that the definition of targets and indicators is often subject to phenomena of “gaming” (Bevan & Hood, 2006; Kelman & Friedman, 2009; Smith, 1995), i.e. the deliberate manipulation of performance systems to secure advantages from the evaluation process by those who define them, e.g. the phenomena called “Ratchet and Threshold” effects described by Bevan and Hood (2006) and Kelman and Friedman (2009). A more theoretical level of analysis points out the need to consider “performance measurement systems as social structures of interaction between individuals and institutions” (Lewis, 2015, p. 8) going beyond an instrumental and rational vision of PM to embrace a more mature vision taking into account the fact that measures, once introduced, have their own autonomy and lead to different perceptions, behaviours and responses from those who deal with them. In this way, PM goes from being a performance detection tool to an instrument influencing the behaviours of organisations and individuals (Espeland & Sauder, 2007; Long, Burton, & Cardinal, 2002).

Alongside this debate on the limits of PM in public administrations, there are other contributions that are based on a pragmatic approach and have sought to identify the factors favouring the development of PM (Angiola & Bianchi, 2015; De Lancer Julnes & Holzer, 2001; Franco & Bourne, 2003; Van Dooren, 2005).

With specific reference to public administration, De Lancer Julnes and Holzer (2001) identify two types of factors that impact on the adoption and implementation of PM systems. From a rational/technocratic point of view, there are four factors favouring the adoption of a performance evaluation system:

• Resources dedicated to measurement from personnel with PM skills.

• Information. The availability of adequate information regarding how PM works.

• Goal orientation. How organisations set goals and strategies for operational purposes.

• External requirements. The availability of requirements that favour the adoption of a PM system, e.g. a legislative provision.

From a political/cultural point of view, the authors highlight three further factors.

• Internal interest groups. The involvement and support of the people working in the organisation under evaluation legitimises changes and favours the use of the results.

• External interest groups and unions. Similarly, the support of those who work outside the evaluated organisation (citizens, trade unions, political representatives) has an impact.

• Risk taking and attitudes. Organisational culture plays an important role in determining acceptance of and change in PM systems.

These are not definitive or exhaustive classifications as they can be expanded with case-specific factors or with the intention of underlining the specific contribution of a factor (Fryer, Antony, & Ogden, 2009; Melkers & Willoughby, 2005; Suppa & Webb, 2016).

A common element among these factors is that they are connected to the PM system and to the organisational conditions in which it is implemented. From a certain point of view, these factors are therefore contingent, i.e. they may or may not occur in relation to actions taken or to the presence of certain circumstances. However, there is a cross-factor, which is within the PM system and is represented by the intrinsic characteristics of the activity under examination. The relationship between the characteristics of the activities and their measurability has been highlighted by a study carried out by Hackman and Oldham (1980) which states that as routines decrease and ambiguity increases, activities are less easily measurable. Blankart (1987), while discussing privatisation and the possibility of implementing it, identifies some services that are more easily measured compared to others for which measurement is more complex. Mascarenhas (1996) classifies all public functions of New Zealand’s public sector agencies according to the degree of difficulty in measuring their outputs and outcomes, thus underlining the importance of adopting different measurement systems based on the characteristics of the activity measured.

In relation to public services, Frey, Homberg and Osterloh (2013), inspired by Ouchi's article (1979), affirm that output control is compatible exclusively with activities characterised by a high level of measurability and attributability of outputs. Outputs must be observable and attributable and not characterised by intensive interdependencies between the various actors involved (e.g. citizens, and other public or private organisations). Otherwise, a PM system is exposed to dysfunctional effects such as "goal of a displacement effect" or "multiple tasking effect". In other words, if there is no way to measure what outputs are produced by the administrative action or if their identification is controversial, the key assumption on which the assessment of performance is based is lost, which inevitably paves the way to dysfunctions such as the measurement of partial or unrepresentative outputs.

A significant contribution in appreciating the relevance of the connection between measurement systems and the activities measured comes from Noordegraaf and Abma (2003) who classifies the activities into three sets on the basis of how much they are known (and therefore subject to strong classifications) and contested (i.e. judged on the basis of shared standards).

Conventional activities are those characterised by routine activities that are well-known and whose standards are uncontested. The outputs of the administrative process can be known, measured and compared over time.

On the other hand, unconventional activities are characterised by non-routine, fuzzy, conflictual and innovative processes. Their outputs can hardly be known, measured and classified. There are no shared standards because they relate to areas with diverging interests and values that change over time. These activities are unstable and make it difficult to measure results against pre-established objectives.

Finally, transition activities are in an intermediate situation between these two cases and refer to “many new and unknown elements of standards that are subject to discussion”.

Abma and Noordegraaf (2003) present a similar reasoning starting from the consideration of the ambiguity that characterises the activities (productive process) carried out by public administrations.

The nature of the interactions between producer and consumers allows us to divide the activities into those aimed at producing a pre-established output through a one-side interaction between producer and consumer and the activities based on a two-side interaction where customers and citizens are co-producers and actively influence the service. In this case, the nature of the production process is characterised by two-side interactions.

The nature of the production process distinguishes between routine or repetitive and standardised production and non-routine production.

Also in this case, the two authors conclude that a strong emphasis on PM is appropriate only in activities characterised by a limited level of ambiguity, i.e. production processes with one-side interactions, and by routines. In other cases, the use of PMs must be carried out with caution, since the ambiguity of the activity being measured reduces the value of the evaluation results (Burgess & Ratto, 2003).

From the NPM perspective, Speklé and Verbeeten (2014), while confirming the link between the activity being measured and the measurement system, stress the importance not only of measuring outputs but also of doing so from pre-established objectives and targets. This need requires not only measurement skills but also the ability to establish targets through the preliminary knowledge of production functions. In this respect, the authors define the contractibility of public sector activities on the basis of their characteristics in terms of ability to set unambiguous objectives, ability to measure outputs avoiding distortions and knowledge of production functions. Only high contractibility activities are compatible with a use of PMs in terms of incentives as emphasised by the NPM programme (Newberry & Pallot, 2004). In other cases (low contractibility), an incentive-oriented use of PM leads to dysfunctional behaviours and distortions in managers’ behaviours. However, for activities with limited contractibility, the exploratory use of PM can be applied in order to identify areas of actions that require special attention or to report priorities or expectations in terms of desired performance.

3. A long path with few results: the introduction of performance measurement in Italian public administrations

The path leading to the introduction of PMs in Italian public administrations is here discussed. The main bodies involved were ministries, as this kind of regulations apply to them directly, unlike other territorial institutions and sectors with specific transposing regulations.

The Italian public administration has traditionally been characterised by an administrative system based on an administrative law with a bureaucratic approach (Capano, 2003; Ongaro & Valotti, 2008). This approach has undergone a major reform since the early 1990s (Ongaro, 2006; Rebora, 1999). One of the most relevant benchmarks concerned the introduction of PM mechanisms (Minelli, Rebora, & Turri, 2008; Ongaro & Valotti, 2008; Rebora, 1999).

The Legislative Decree 29/1993 is the first measure that provides for the internal evaluation of the outputs of the activities carried out by public administrations. In this measure, it is easy to spot a NPM-based approach: the definition of objectives to be achieved and their evaluation by the political management bodies is the prerequisite for the full managerial autonomy of the managers. However, a subsequent law (Legislative Decree 286/1999) defines the terms of the PM activity, entrusting it with the task of estimating the congruence between the outputs obtained and the pre-established objectives. For this purpose, the law provides that the offices and the subjects in charge of evaluation activities are to be called “Internal Control Offices” (Secin) and to issue, at least annually, a report reserved to the minister on the results of their analyses, with proposals for improvement.

Although the legal obligation encourages (or forces) all ministries to implement ways to measure performance, results have been very modest (Hinna, Meneguzzo, Mussari, & Decastri, 2006; Minelli et al., 2008; Turri, 2007). Control activities are limited to verifying the achievement of pre-established objectives in terms of compliance. All of this occurs without documents summarising the activities that have been carried out and without further use of the obtained results.

Ten years later, the failure of the system outlined by Legislative Decree 286/1999 led to the reform of the PM with Legislative Decree 150/2009. As acknowledged by the promoters themselves, this reform also has a NPM-based approach (Department of Public Administration, 2009). The performance assessment system (called “performance cycle”) is based on two documents that each PA organisation is obliged to adopt. The first, The Performance Plan, is a preliminary programmatic document to be adopted annually by 31 January by a political body, which identifies the strategic and operational objectives and related indicators that are supposed to guide administrative actions during the evaluation exercise. The Performance Report, which is always approved by political bodies, is a final document reporting the degree of achievement of the objectives and highlighting possible shortcomings.

Both Legislative Decree 286/1999 and Legislative Decree 150/2009 provide for measures to link part of the remuneration of the personnel working in public administrations to the results achieved in a performance-related pay system. This measure applies in the first place to managers and employees in charge of offices. In particular, Legislative Decree 150/2010 closely links the measurement of organisational performance (i.e. the outputs obtained by the public administration and its operations) with the measurement of individual performance (i.e. the evaluation of the results obtained by employees in a performance-related pay system). The intention is to encourage “the quality of work” and “the recognition of merits and demerits” (art. 1, Legislative Decree 150/2009).

From a methodological point of view, this legislative measure is based on three main points:

• general PM for all administrations and all their activities without distinction;

• the request to highlight and measure the achievement of “specific and measurable objectives in concrete and clear terms”, in relation to a limited time span and “objectives that are based on reference values derived from pre-established standards” (art. 5, Legislative Decree 150/2009);

• the introduction of a series of measures to encourage the implementation the reform in organisations, such as the creation of a national agency for the promotion of PM and the reconfiguration of PM units called OIV—independent assessment body in each administration (for a more complete list, see Table 1, which is based on the classification of De Lancer Julnes & Holzer, 2001).

Table 1. Interventions to favour performance measurement in Italian public administrations

	Provisions of Legislative Decree 150/2009	Elements added in 2017
Rational/Technocratic point of view
Staff with measurement skills and information on performance measurement	The Independent Assessment Body (in Italian OIV) is required to have a high level of professionalism Creation of a national agency to accompany the reform	OIV members only enrolled from the national register (selective criteria for enrolment and obligation of continuous training) Reconfiguration of the national agency to accompany the reform
Goal orientation	The entire process of performance measurement is aimed at verifying the objectives	Strengthened ability of the National Agency to set out guidelines for the performance cycle
External requirements	When documents of the performance cycle are absent performance bonuses and staff hiring are prohibited to the defaulting administrations	The mandatory nature of the performance cycle is reaffirmed and strengthened
Political/Cultural point of view
Internal interest groups	Attempt to implement incentives to involve the staff Approval of key documents by elected bodies
External interest groups and unions		Participation of trade unions in the definition of the criteria for the differentiation of evaluations (art. 19) and participation of citizens in the measurement of performance (art. 19-bis)
Risk taking and attitudes	Introducing a top-down change in the way public administration works by steering it towards results	–

In general, law enforcement, supported by legislation and accompanied by enforcement measures, is widespread, but it is difficult to achieve the objectives for which it was adopted. The PM documents have hardly become instruments wherewith the political authorities assign objectives and verify their results. These documents are mainly descriptive and are limited to providing contextual information on the activities of administrative bodies but do not take into account the guidelines and, especially, do not report whether objectives are achieved (Barbato & Turri, 2017). Despite the fact that the 2009 Legislative Decree provided for the classification of individual performance assessments into four groups, which were then cancelled by other regulatory measures to contain public spending, the measurement of individual performance produced outputs that were not very differentiated and levelled upwards (Ongaro & Valotti, 2008; Rebora, Ruffini, & Turri, 2017). Thus, the link between PM and performance-related pay is reduced to a means to keep distributing the result-based remuneration indiscriminately rather than becoming a management tool to encourage learning (Barbato & Turri, 2017; Capano, 2003; Turri, 2007).

Following the acknowledgement of these difficulties in May 2017, the Legislative Decree 150/2009 was amended with some measures that, while maintaining the previous system unchanged, aim to strengthen (see Table 1):

• the link between PM and incentive measures;

• the organisational mechanisms for the implementation and support of the reform.

4. Research design and data

On the basis of what has been illustrated in Section 2, the expectation is that, in the presence of a drive for the introduction and development of PM systems in the public administration, with the same factors favouring the development of such systems being equal, some activities are measurable in terms of performance while others are measurable to a lesser extent. In this regard, on the basis of the classification of Abma and Noordegraaf (2003), a distinction is made between highly ambiguous and less ambiguous activities. Highly ambiguous activities are those characterised by two-side interactions between producers and consumers, where customers and citizens are co-producers and actively influence the service and the nature of the routine production process. Conversely, low ambiguity activities are intended to produce a predefined output through a one-side interaction between producers and consumers by introducing a routine production process.

This paper sets out to explore these aspects starting from the analysis of the objectives—indicators whereby performance is measured in the 11 Italian ministries in 2016. The objectives–indicators, which totalled 1,511 as showed in Table 2, are included in the performance plans of the ministries. In order to facilitate the analysis, objectives–indicators have been listed in a working document together with available information in the documents examined (performance plan) in relation to reference organisational units, target and formula for calculating such indicators.

For the reasons mentioned in Section 3, Italy is an interesting case for the following aspects: a multiannual effort to implement PM systems, a progressive refinement of the conditions favouring the development of these systems. But above all, the Italian case presents a PM-based experience supported by binding measures, even accompanied by sanctions for non-compliance, which is paradigmatic as it highlights the limitations of an implementation that does not take into account the measured activities and the distortions that this choice entails. The choice to explore this phenomenon at the level of ministries is due to the fact that the law that makes PM mandatory is automatically implemented in ministries, whereas in other sectors, it was applied more gradually. Moreover, the ministries represent a relatively homogeneous sector at least in terms of exposure to factors that favour the development of PM systems.

In a NPM perspective and assuming a definition of performance as a production process in which performances are the output of activities (Van Dooren et al., 2010 also in relation to the limits of these definitions), the ability of each indicator to measure the output of a certain activity was verified. The distinction between output and outcome has not been taken into consideration at this stage (for further discussion on this subject, see Wilson, 1989). It was thus possible to distinguish between performance indicators (output control) and process indicators (process control). With a managerial approach, the first type of indicators focuses on the results of the administrative action, not on the process and the behaviours that necessary to achieve them (Eisenhardt, 1985; Frey et al., 2013). The second type is influenced by the bureaucratic administrative model (Weber, 1978) and does not focus on the output but on the performance of administrative activities by verifying the compliance with rules, procedures and obligations.

For the output indicators, the relevance of the outputs detected in relation to the activity in terms of representativeness was ascertained below with an additional examination. The output indicators were therefore examined on the basis of their ability to effectively measure the achievement of relevant results in relation to the administrative activity carried out. The literature has long highlighted the tendency of measurement systems to focus on measurable aspects at the expense of unquantified aspects of performance (Smith, 1995 in relation to tunnel vision and measure fixation), thus providing information that is quantifiable but not relevant to understand the real performance of the organisation as a whole. Moreover, this trend has been pointed out precisely with reference to the spread of PM in Italian public administrations (Chevauchez, 2014; Rebora et al., 2017).

Operationally, the simultaneous presence of two requirements was jointly verified. The first requirement is the reference of the indicator to an activity representing the administrative action carried out, providing a relevant representation of the output achieved. This review was carried out starting from the analysis of the activities carried out by the organisational units through the examination of the tasks assigned to them, which were determined by consulting official documents (financial statements, explanatory notes) and their websites. At the same time, the ability of an indicator to measure the output or to provide contributions to quantify the resulting performance was verified, also in comparison with expected targets. The notion of measurement is quite complex. In empirical sciences, certain conditions are required for measurements, more precisely “assessments with an intersubjective and objective character” (Mari, 2003). In essence, it is necessary that measurement results provide different observers with the same information; at the same time, measurement results must provide information only in relation to the measured object and not to its environment, which may also include the observer; therefore, measurements of the same object under different conditions and in the presence of different observers must produce the same results (Mari, 2003). In this analysis, in the absence of more detailed information, the examination was aimed at verifying the ability to quantify a performance by studying the measuring system of the indicator under examination.

The verification of this combination of relevance and performance measurability has allowed the identification of a subset of indicators that can be properly defined as output indicators.

Subsequently, for each of the 1,511 objectives–indicators, the activities carried out by the organisational units to which the objectives–indicators are assigned were considered in order to ascertain the degree of ambiguity (Abma & Noordegraaf, 2003) of the activity to which the indicator refers. The degree of ambiguity was analysed by examining two aspects. The first aspect is linked to the routine or non-routine nature of the production process and is therefore associated with the degree of standardisation and repetitiveness of the processes. The second aspect is the importance of the action of third parties in determining and influencing the outcome of the production process. On the basis of these aspects, a binary classification has been made between indicators linked to highly ambiguous activities and indicators linked to less ambiguous activities.

Table 3 summarises the phases of the classification of the objectives/indicators under scrutiny.

Table 2. Objectives and indicators in the performance plan

Ministry	No. indicators	No. indicators
Foreign Affairs	98	From 14 to 111
Cultural Heritage	188	From 212 to 299
Labour and Social Policy	77	From 300 to 376
Environment	96	From 377 to 472
Economy and Finance	225	From 473 to 794
Interior	102	From 9 to 13 and from 478 to 785
Education, Universities and Research	156	From 795 to 950
Defence	48	From 951 to 998
Justice	71	From 1 to 8 and from 999 to 1061
Health	52	From 1062 to 1113
Infrastructure and Transport	152	From 1114 to 1265
Agricultural Policies	41	From 1266 to 1306
Economic Development	205	From 1307 to 1511
Total	1,511

Table 3. Analysis phases and classification of objectives/indicators

Phase	Description	Primary references
1	Classification of objectives/indicators into Output control focuses on the results of the administrative action, not on the process and the behaviours that necessary to achieve them Process control not focus on the output but on the performance of administrative activities by verifying the compliance with rules, procedures and obligations	Weber (1978); Ouchi (1979); Eisenhardt (1985); Frey et al. (2013)
2	Classification of objectives/indicators into • Relevant output indicator: 1. Reference to an activity representing the administrative action carried out, providing a relevant representation of the output achieved 2. Good performance measurability• Not relevant output indicator1. Reference to a marginal activity, providing a not relevant representation of the output achieved2. Low-performance measurability	Smith (1995); Mari (2003); Caglio and Dittilo (2008); Frey et al. (2013); Chevauchez (2014); Speklé et al. (2014); Ditillo et al. (2015); Rebora, G., R. Ruffini and M. Turri (2017)
3	Classification of objectives/indicators into• highly ambiguous activities:1. Not routine activity (weak degree of standardisation and low repetitiveness of the processes)2. Two-side interactions between producers and consumers, where customers and citizens are co-producers and actively influence the service and the nature of the routine production process• Low ambiguous activities:1. Routine activity (strong degree of standardisation and high repetitiveness of the processes)2. Produce a predefined output through a one-side interaction between producers and consumers by introducing a routine production process	Ouchi (1979); Abma and Noordegraaf (2003); Speklé et al. (2014)

By cross-checking the nature of the indicator (process or output) and the nature of the activity to which it refers, the matrix below is produced (Figure 1).

The examination and classification of the objectives–indicators was carried out in advance separately by each of the three researchers after sharing the criteria. The researchers then compared the divergent results and reached a common position after a detailed discussion on the subject.

5. Results and discussion

The analysis of the PM system in the Italian case was based on the detailed examination of the 1,511 indicators and the related objectives in the 11 Italian ministries (phase 1 in Table 3). First of all, the nature of the indicators was examined and it was found that only 25% (376 out of 1,511) are output indicators whereas process indicators (1,035 out of 1,511) still prevail.

On the basis of these data showed in Table 4, we can confirm what the literature has already highlighted in relation to the Italian case, i.e. the difficulty of the PM system to detect the performance of administrative activities in terms of output and the resilience of the traditional administrative model based on bureaucratic process controls (Bonini Baraldi, 2014; Capano, 2003; Ongaro & Valotti, 2008; Rebora et al., 2017). It should be noted that the main purpose of a PM-based system is to measure performance and not the production process.

Instead of measuring the outputs of the ministerial activity, the control system ensures that the procedures for the drafting of documents have been followed (drafting of the technical specifications of the tender for resources intended for rescue in historic centres #782) and verifies the performance of administrative activities (incoming certified email messages processed through an internal management system; outgoing certified email via PEC #600) and in general the ability to meet regulatory requirements (degree of adoption of the implementing measures required by legislative provisions #1027). A large number of indicators and targets are in relation to the compliance with the deadlines of resource payments and transfers and the degree of usage of the resources allocated to the offices. In this perspective, cost containment policies are interpreted as a way to cut costs which is not connected with the achievement of results (reduction, in percentage terms, of postal expenses of the government’s territorial offices and the police in 2016 compared to the expenses incurred in 2015 #794; reduction of programme-related costs relating in the previous year #1109). Thus, process controls, which are rejected by the NPM because of their bureaucratic approach, are in the limelight again. This trend benefits from the existence of public expenditure containment measures and economic measures which, if interpreted in a top-down and legalistic way, are not connected to the intention of producing cheaper outputs but simply to cut costs without taking care of the consequences in terms of output. Despite the rhetoric of managerialism, the bureaucratic paradigm is still hegemonic and capable of annihilating managerial pressures.

The difficulty in developing output indicators is even more evident by focusing on the link between objectives—indicators, activities and the organisational units—to which the indicators are associated. Focusing on the 376 output indicators, it can be noted that only a subset, 140 indicators (37%), can provide relevant measurements with respect to the activity to which they refer (phase 2 in Table 3). In other cases, even though the indicator is connected to a generic output, this is of little relevance for the achievement of the pre-established objective or for the institutional objectives of the organisational unit to which the indicator is associated. In many cases, in fact, the activity under examination is reduced to the possibility of organising visits or meetings (bilateral and multilateral visits or meetings #44), to the number of projects carried out (implementation of environmental communication projects #394) or to the number of checks carried out (verification of safety management systems in the workplace #776). Each indicator lacks the ability to represent and quantify whether these activities have a real result in terms of achievement of the pre-established institutional objectives. Thus, although the system actually measures the achievement of a result, it is not representative of the activities carried out by the organisational units under examination.

The presence of a large number of quantitative indicators, which are however not very representative of the whole objective and consequently of the public activity of reference, suggests both a case of “measure fixation”, in other words, when a performance measure does not fully capture all dimensions of the associated objective (Smith, 1995, p. 290), and a case of “tunnel vision”, in which emphasis is given on phenomena that can be easily quantified, at the expense of unquantified aspects of performance (Smith, 1995).

The overall figure of relevant output indicators (140 indicators out of 1,511, less than 10%) mainly confirms the failure of the reform (see Table 5). Despite the legislative requirements, the reform has only been ceremonially implemented and has not changed the way organisations operate (Fattore, Iacovone, & Steccolini, 2018). The basic objective of a PM system is to measure the activities produced. The data collected and analysed show that the reform has failed in its purpose. Hence the question: What are the causes of this failure? Several studies have investigated the reasons underlying the functional difficulties of the Italian PM system. These reasons include the relevance of the national context and its administrative tradition (Capano, 2003; Cerase, 2017), the need to consider the reactivity to measurement by taking into account the reaction of the evaluated subject to evaluation, which may result in not very challenging indicators (Barbato & Turri, 2017; Corte dei Conti, 2012; Micheli & Neely, 2010) and the prevalence of top-down methodological indications that impose undifferentiated performance detection methodologies that do not take into account specific organisational contexts (Rebora et al., 2017).

Section 3 has illustrated the legislator’s tendency to introduce factors that favour the implementation of PM at a rational/technical and political/cultural level. An effort has certainly been made both with the 2009 reform, which nevertheless appears to have produced modest results, and with the 2017 reform, the effects of which cannot yet be seen on the basis of the data. If the factors identified above undoubtedly play a role, there is another and different transversal factor, endogenous to the PM system, which must be considered: the intrinsic characteristics of the activity being measured in terms of ambiguity.

A first analysis in this regard shows how out of 1,511 indicators considered, 743 (49%) are related to ambiguous activities or low level of routine and two-side interactions, whereas 768 (51%) indicators are associated with activities characterised by low ambiguity or high level of routine and one-side interactions (phase 3 in Table 3). This figure must be considered with caution. In fact, it is not possible to conclude that the proportion between indicators referring to ambiguous activities and unambiguous activities is representative of the real division of activity in the ministries. In fact, the data observed in Table 6 do not provide any assurance with regard to the real distribution of more or less ambiguous activities in the ministries but only allow ascertaining characteristics in terms of ambiguity of the activities to which the indicators refer.

By crossing the characteristics of the indicators in terms of output/process and in terms of the level of ambiguity of the activities to which the indicators refer, the following matrix is obtained. As regards the output indicators, the overall value and the value referring to the actually relevant indicators are reported.

The first quadrant shows process indicators associated with highly ambiguous activities. This is the largest set of indicators. This is why, for activities which are certainly ambiguous, such as “managing relations with the EU in the field of development cooperation” or “promoting political issues relating to international bodies and fora”, the Ministry of Foreign Affairs uses process indicators such as “time taken to pay European Development Fund shares” (#24) or “number of meetings within international organisations, forums and other venues, in which participation was assured” (#60). The second quadrant shows process indicators associated with limitedly ambiguous activities. The Ministry of Cultural Heritage and Activities and Tourism presents indicators such as “number of adopted requirements/total requirements to be adopted” (#132) and “timeliness of payments” (#179). Quadrants III and IV show both the overall data and the data referring to the relevant indicators (in brackets). The third quadrant shows the number of activities characterised by a low level of ambiguity and output indicators. The Ministry of Justice collects performance information using indicators such as “evasions from prisons and during transfers” (#1005) and “number of prisoners who have attended a course in literacy and school education” (#1020). The Ministry of the Environment uses indicators such as “increase in waste sorting” (#382). Finally, the fourth quadrant presents 27 significant output indicators referring to highly ambiguous activities, which are shown in Table 7.

Table 4. Process and output indicators and ministries

Ministry	Output indicators	Process indicators
Foreign Affairs	20	78
Cultural Heritage	20	168
Labour and Social Policy	21	56
Environment	29	67
Economy and Finance	12	213
Interior	18	84
Education, Universities and Research	29	127
Defence	16	32
Justice	21	50
Health	18	34
Infrastructure and Transport	106	46
Agricultural Policies	17	24
Economic Development	49	156
Total	376	1,135

Table 5. Output indicators including ministry and distinction between relevant and not relevant indicators

Ministry	Output indicators	Of which relevant	Of which not relevant
Foreign Affairs	20	4	16
Cultural Heritage	20	12	8
Labour and Social Policy	21	9	12
Environment	29	12	17
Economy and Finance	12	9	3
Interior	18	6	12
Education, Universities and Research	29	17	12
Defence	16	14	2
Justice	21	19	2
Health	18	4	14
Infrastructure and Transport	106	13	93
Agricultural Policies	17	4	13
Economic Development	49	17	32
Total	376	140	236

Table 6. Indicators related to low/high ambiguity activities by ministries

Ministry	Low ambiguity activities	High ambiguity activities
Foreign Affairs	34	64
Cultural Heritage	110	78
Labour and Social Policy	12	65
Environment	24	72
Economy and Finance	40	185
Interior	82	20
Education, Universities and Research	121	35
Defence	43	5
Justice	67	4
Health	21	31
Infrastructure and Transport	151	1
Agricultural Policies	13	28
Economic Development	50	155
Total	768	743

Table 7. List of relevant output indicators referring to ambiguous activities

Immigration and entry visa requests to Italy processed within 25 days	57
Occurrence of extreme poverty	#301
Percentage of children aged 0–3 who have accessed childcare services (crèches, daily nurseries or innovative and integrative services), out of the total population aged 0–3	#302
degree of coverage of the survey of the services of the EURES Advisers	#356
Effectiveness of dispute resolution processes	#358
Mapping databases of public sector bodies for communication to the European Union	#433
Coverage of operating costs of Carabinieri Corps for the protection of environment	#466
Reduction of risk by stabilising/extending the average life of debt	#579
Coverage of public debt securities auctions	#580
Organisation/participation in joint return charter flights under the coordination of FRONTEX	#628
Average time to send the device to parts	#706
Planning and defining the teacher evaluation system	#837
Rate of increase of first level enrolments in scientific disciplines compared to the previous academic year	#872
increase in the number of sheep participating in the plan (subsequent years)	#1275
number of farms participating in the plan to control and eradicate IBR in cattle	#1277
Shows the percentage variation of offences in the area	#1284
Increase in the volume of exports in the agri-food sector compared with the average for the previous three years	#1291
Percentage of increase of the gas transportation network through methane pipelines (authorised km on request of operators)	#1325
Optimising the use of frequency resources	#1375
Reduction of the wholesale price gap of electric energy compared with EU countries	#1393
Variation in the impact of fossil fuels on primary energy consumption	#1394
Annual trend in car insurance rates	#1401
Growth rate of enterprises	#1422
Increase of value of exports to signatory countries to free trade agreements with the EU	#1447
Growth rate of Small Medium Enterprises	#1454
Growth rate of Italian FDI abroad	#1491
Growth rate of Italian exports	#1493

Table 8. Subdivision between output and process indicators in relation to activities with a low/high level of ambiguity for each ministry

Ministry	L P	H P	L O*	H O*	Overall indicators
Foreign Affairs	30	48	4 (3)	16 (1)	98
Cultural Heritage	93	75	17(12)	3 (0)	188
Labour and Social Policy	5	51	7 (5)	14 (4)	77
Environment	14	53	10 (10)	19 (2)	96
Economy and Finance	33	18	7 (6)	5 (3)	225
Interior	71	13	11 (5)	7 (1)	102
Education, Universities and Research	105	22	16(15)	13 (2)	156
Defence	29	3	14(14)	2 (0)	48
Justice	46	4	21 (19)	0	71
Health	15	19	6 (4)	12 (0)	52
Infrastructure and Transport	45	1	106 (13)	0	152
Agricultural Policies	8	16	5 (0)	12 (4)	41
Economic Development	31	125	19 (7)	30 (10)	205
Total	525	610	243 (113)	133 (27)	1,511

L: Low ambiguity; H: high ambiguity; P: process indicators; O: output indicators.

*Number of significant indicators in brackets.

By examining the distinction of indicators in the four quadrants, it is possible to verify that output indicators are more widespread among activities with limited ambiguity. As regards the indicators referring to activities with low ambiguity, 28.9% are output indicators and 16.3% are actually relevant output indicators. As far as ambiguous activities are concerned, the percentage of output indicators is 21.5%, whereas the percentage of relevant output indicators even drops to 3.3%. The limited presence of output indicators relevant to highly ambiguous activities does not occur only overall but in each ministry with the partial exception of the Ministry of Economic Development (10 relevant output indicators referring to ambiguous activities).

Overall, the data collected in Table 8 show that in the case examined, the difficulty in measuring ambiguous activities is greater compared to unambiguous activities. The data therefore confirm the presence of a relationship between administrative activity and the type of measurement that can be applied to it. When this link is not taken into account, as in the present case, where regulatory requirements and methodological indications do not distinguish on the basis of the activities carried out, there are consequences.

First of all, some activities (in particular the unambiguous ones) can be more easily measured in terms of output (Figure 2, quadrants II and III), in particular when relevant output objectives are considered.

Figure 1. Matrix for the examination of the degree of diffusion of PM in relation to the type of activity detected.

Figure 2. Indicators in Italian ministries according to activity type.

Second, the compelling drive to introduce measurements without emphasis on the activity being measured favours the diffusion of irrelevant indicators, thus resulting in paradox performance, i.e. the presence of indicators that focus on process or marginal aspects of administrative activities and thus result in an evaluation that does not match with the actually achieved level of performance (Van Thiel & Leeuw, 2002). In particular, the use of process indicators leads to “sub-optimisation” or “myopia” (Smith, 1995), which means that the focus is on specific aspects of the production process while neglecting the overall process (Van Dooren et al., 2010). Moreover, the use of non-relevant result indicators leads to issues related to measure fixation and tunnel vision (Smith, 1995) as illustrated above. In these circumstances, particular attention is paid to those aspects of administrative activities that can be easily quantified even if they are sometimes marginal with respect to the achievement of results based on a “more is better” mentality, whereby emphasis is placed on the production of some outputs without regard to the actual quality of the production process (Bohte & Meier, 2000).

In this way, the actual ability of a measurement system to affect the performance of public administrations is eroded. This difficulty in the measurement system also leads to the occurrence of episodes of deliberate paradox performance or gaming by the actors involved in order to maximise their usefulness (Frey et al., 2013; Van Thiel & Leeuw, 2002). It is precisely the mismatch between indicators and the actual output measurement that leads to the emergence of opportunistic behaviours. This scenario cannot be supported by the data of this work; this tendency has however been reported by other studies on the Italian case (Barbato & Turri, 2017; Corte dei Conti, 2012; Rebora et al., 2017).

All of this, moreover, condemns the PM system to a structural under-representation of indicators and related ambiguous activities, given the difficulty in identifying output indicators, especially when they are relevant. Essentially, some activities are not as exposed to the spotlight of PM as others because they are less prone to be detected. The resulting risk is that measurement systems produce a false representation of the activities, thus creating, as stated by Power (1997), a fictional representation disconnected from the production process. The purpose of this representation is essentially ceremonial, as it meets the need for external validation (Brignall & Modell, 2000).

The examination of the 27 output indicators associated with ambiguous activities offers, at least in an exploratory way, some interesting insights. First of all, the majority of indicators do not focus on the nature of the outputs produced by the productive process and measure the trend of the social phenomenon underlying the administrative action. A certain amount of indicators of this type can be found at the Ministry of Economic Development. Measures such as “incidence of absolute poverty” (#301), “percentage variation of crimes in the area” (#1284) and “growth rate of SMEs” (#1454) do not measure the output of administrative action but focus on the social phenomenon related to it. The connection between administrative action and social phenomenon is very complex, certainly a two-side interaction connected to environmental factors that are beyond the control of the administration. Although it can be expected that, in such contexts, the direct connection of the PM system with performance-related pay mechanisms may generate resistance and favour gaming, the collected data show that there is room for an alternative use of the PM system. The indicators can be used to explore the sense-making process of PM (Speklé & Verbeeten, 2014) in order to identify areas of actions that require particular attention, to indicate priorities or expectations in terms of desired performance and to draw the attention of the organisation to the expected aims of public action.

6. Theoretical and managerial implications

On the basis of non-episodic data from to an entire sector of Italian public administration, this study has highlighted a link between the spread of PM, i.e. output control, and the type of activity detected in terms of higher or lower ambiguity. Failure to consider this link is also linked to the presence of unrepresentative and paradoxical performance indicators.

On this basis, it is possible to make two final considerations—one technical and one theoretical—which can shed light on the link between administrative activities and measurement. The two following considerations are limited by the fact that they are drawn by the examination of a specific reality: the PM system of the Italian ministries. Their value in other countries and in other public sectors may be corroborated by future studies.

The first concerns the issue of measuring the activities of public administrations. This study confirms that there is a marked and deep-rooted delay in the culture of public administrations in the approach to PM. Public administrations are still stuck with a metaphysical conception of measurement, whereby it is believed that it must be aimed exclusively at measuring reality and struggles to evolve towards an antimetaphysical conception, i.e. measurement as a means of representing reality, and towards a relativistic conception that gives prominence to project intention (Micheli & Mari, 2014; Rebora et al., 2017). Considering measurement as a technical and neutral tool impoverishes PM by imposing an undifferentiated approach to the different activities which, on the other hand, can be measured with different approaches, as demonstrated in relation to their level of ambiguity. This view does not allow us to take into account the phenomenon of reactivity and the natural propensity of the controlled to interpret and change their behaviour on the basis of control (Espeland & Sauder, 2007; Lewis, 2015). This is a very significant issue. A more evolved view on measurement is connected to a view of the PM system that does not aim at ascertaining the truth but has the purpose of steering behaviours and results towards the desired goals by adopting the most appropriate models and methodologies. The case under examination shows that this transition has not yet taken place and reveals the limits of an unconscious application of PM instruments.

The second consideration is part of the debate on the introduction of the NMP in public administration and the subsequent difficulties. NMP-based reforms have been widely implemented in Italy, as demonstrated by their spread driven by binding measures, but this implementation was only superficial, as demonstrated by the prevalence of process control. The examination of the factors leading to the introduction of PM in Italy shows that there was an effort to accompany PM, albeit with a certain delay. The weakness of the system seems to be linked not only to underestimation at both technical and operational level but also to a weak theoretical and methodological elaboration. This pedantic application mainly underestimated the intrinsic characteristics of NPM and PM. In his 2003 textbook The Essential Public Management, Pollitt had already observed this issue by explaining that the success of NPM-based reforms is closely linked to the degree of publicness of the activities being reformed.

(…) many of the elements of NPM are drawn from private sector, so they should more easily fit functions which are closer to the private sector in terms of their base characteristics. (Pollitt, 2003, p.50–51)

The link with the activities is not accessory (Fountain, 2001). Conversely, it is essential for the success of a reform and of the PM in particular, as the literature argues and this study demonstrates. In Italy, not only was this link not taken into account, it has never been on the agenda in the first place. This confirms a mainly nominal and mainly superficial approach that must be faced not only on the operational level by implementing new solutions or supporting factors but also primarily on the theoretical level with deeper reflection on various aspects including functions, limits and possibilities of the instruments used. With this awareness, it is possible to extend the use of PM to functions and activities to which it can hardly be applied, by enhancing its exploratory value and its ability to direct measurement.

Additional information

Funding

The authors received no direct funding for this research.

Notes on contributors

Giovanni Barbato

Giovanni Barbato is a PhD student in Economic Sociology at the Department of Social and Political Studies of University of Milan. His research interests focus on performance management systems and strategy of public organisations, in particular higher education institutions.

Alessandro Salvadori

Alessandro Salvadori is a research fellow at the Department of Management, Economics and Industrial Engineering at Politecnico di Milano. His research interests focus on performance evaluation in public organisations, e-government and digital agenda.

Matteo Turri

Matteo Turri holds a PhD in Management. He is presently an associate professor in Public Management in the Department of Economics, Management and Quantitative Methods (DEMM), University of Milan. Turri’s research and relevant scientific production focuses on evaluation and control systems in public administration and performance management in public sector organisations.