Priorities and targets: supporting target-setting in healthcare

Abstract

Management by objectives requires selecting the appropriate number of indicators to measure objectives and then defining high-priority indicators. Failing to address these two issues often results in the so-called ‘performance paradox’. This paper describes an algorithm applied in the healthcare sector in the Italian regions. The resulting performance evaluation system is able to detect priority indicators in the target-setting phase, improving management and saving costs.

Keywords:

• Healthcare
• management by objectives
• performance evaluation
• performance paradox
• priority-setting

Governance based on planning and control systems with performance indicators (PIs) and targets is a form of indirect control that is widely deemed necessary to administer any complex system (Beer, 1966; Pollitt, 1986; Carter et al., 1992; Bird et al., 2005). However, managing by objectives requires tackling at least two difficult issues:

• Defining the appropriate number of indicators.

• Choosing a rigorous principle to define which indicators should be considered as high priorities.

Regarding the first point, the process of governance by PIs and targets relies on a set of measures representing social complex phenomena. This translation process—from social phenomena to quantitative measures— requires careful design. Only some of these phenomena can be represented by quantitative figures, because indicators portray the ‘measurable’ part of the observed object.

The process of governance by PIs and targets relies on the assumption that available indicators are both representative and accurate. The so-called ‘synecdoche’ principle assumes that the measurable part will satisfy both these conditions, by accurately representing the whole object (Carter et al., 1992). However, measurement systems almost never comply with the synecdoche principle, because the set of selected indicators is rarely able to portray the complexity of the evaluated processes (Bevan and Hood, 2006) and may cause distortions and stimulate undesired behaviours on the part of the agents who are controlled by the system.

Studies on the distorting effects of control systems are extensive (Birnberg et al., 1983; Briers and Hirst, 1990; Le Grand and Bartlett, 1993). The basic point is that, in order to make the planning and control scheme meaningful, the measures used should be consistent with the performance targets (which should, in turn, be representations of organizational objectives) so that the behavioural responses of the agents are matched to the requirements of the principal (Smith, 1995). The selection of the indicators and the consequent target-setting phase are therefore crucial, since they can potentially generate incoherent behaviours in the system and lead the organizations towards undesired goals (Ferreira and Otley, 2009; Locke and Latham, 2013).

Since the 1950s, scholars have been studying the dysfunctional consequences of measurement; many of them have looked at ‘output distortions’, i.e. attempts to achieve single targets at the cost of significant but unmeasured aspects of performance (Ridgway, 1956; Nove, 1958; Miller, 1993; Kornai, 1994; Smith, 1995; Heinrich, 2002; Hood, 2002; Propper and Wilson, 2003; Bird et al., 2005; Bevan and Hood, 2006).

In 1991, Bouckaert and Balk described ‘13 diseases’ of public productivity measurement, which resulted from wrong assumptions and problems concerning the content, position and amount of measures (Bouckaert and Balk, 1991). Similarly, Smith wrote about eight potential and unintended behavioural consequences of performance data use (Smith, 1995). These problems are all the result ignoring the synecdoche principle, which causes an inconsistency between agents’ goals and the actual goals wanted by the principal: ‘tunnel vision’, ‘suboptimization’, ‘myopia’, ‘measure fixation’, ‘misrepresentation’, ‘misinterpretation’, ‘gaming’, and ‘ossification’ (Hopwood, 1973; Smith, 1993, 1995; NHS Executive, 1999, 2000; Goddard et al., 2000; 2002). Tunnel vision is when management emphasizes quantifiable phenomena in a performance measurement scheme, at the expense of unquantifiable or unmeasured aspects of the performance.

Since the 2000s, many scholars have been investigating the reasons behind failures of results-based management, with a particular focus on the public sector (Hofstede, 1981; Eden and Hyndman, 1999; Van Thiel and Leeuw, 2002).

Managing by objectives requires tackling two issues: first, defining the appropriate number of indicators that work as objectives; second, choosing a correct principle to define which indicators should be considered as high priorities. Regarding the second point, some recent studies have found that public managers prioritize competing performance goals according to political control mechanisms (Rutherford and Meier, 2015). This means that the determinants of managerial priorities are not always to improve performance or produce better value for money, but are the result of bureaucratic control/hierarchical pressure.

The result of all these limitations is the so-called ‘performance paradox’, which is a weak correlation between PIs and performance (Meyer and O’Shaughnessy, 1993; Meyer and Gupta, 1994). Certain characteristics of public sector organizations can increase the chance of a performance paradox occurring (Le Grand, 1991; Torenvlied, 2000; Fountain, 2003). Performance measurement systems in the public sector can degenerate into formal ceremonies that have little impact on behaviours and do not contribute to improve the efficiency, effectiveness and accountability of an organization (Sharifi and Bovaird, 1995).

In countries that have adopted universal healthcare coverage financed by general taxation (Morton and Cornwell, 2009), profit is not significant while the concept of ‘value for money’ is very important. Performance evaluation systems in these cases need to include a wide range of indicators (Taylor and Taylor, 2015). Process, efficiency and productivity indicators should be included in the performance evaluation systems (Murray and Frenk, 2000; Arah et al., 2003; 2006). A key issue is the choice of the appropriate number of indicators to use—too many targets and indicators might disorient and generate confusion and then priority actions can be ignored (Van der Stede, 2009). On the other hand, the fewer the number of PIs, the more difficult it is to obtain an accurate performance overview (Meyer and Gupta, 1994). Indicators also need to change over time (Van Thiel and Leeuw, 2002, p. 276).

Van Thiel and Leeuw (2002) point out the need to find a balance between an expansion of the number of performance measures on the one hand, and a reduction of the measure pressure on the other. The target-setting phase should, therefore, include a number of targets large enough to limit the synecdoche and ‘output distortion’ problems and, at the same time, sufficiently small to keep the controlled actors focused on the priority issues.

Goddard proposed selecting some ‘headline’ indicators—focused on priority issues that are published and used to hold organizations to account. Other indicators would be used for internal benchmarking purposes only (Goddard et al., 2002). Smith suggests developing a dynamic performance evaluation system that would adapt the choice of the ‘right’ indicators to the situation (Scriven, 1973; Smith, 1993, p. 141).

The methodology we describe in this paper reflects both proposals, by combining the completeness of a wide performance evaluation system (PES) with a rigorous and iterative selection of a few headline indicators. The algorithm we describe sets regional priorities and selects a subset of indicators for the target-setting phase that will reflect the complexity of the healthcare systems without creating confusion with too many indicators.

We developed our PES to support Italian regional policy-makers in the target-setting phase and it is being used by the Italian regions. Our system provides regions with about 200 indicators (see, for details, Nuti et al., 2015). The Laboratorio MeS developed the performance evaluation framework as an independent research unit and co-ordinates and manages information-sharing and data acquisition. The 14 regions in the network have agreed on the criteria to calculate the indicators. Each region is responsible for processing its own data in order to increase the awareness and expertise of regional managers and their staff. The PES measures the results with a multidimensional approach in quantitative terms and then assesses the performance of 100 indicators out of 200: excellent, good, sufficient, poor, or very poor. These five evaluation tiers are associated with different colours, from dark green (excellent performance), to red (poor performance). Regions use the same reference standards for their evaluation, based on academic literature, national standards or, when these are lacking, on the median of the 150 health authorities (HAs).

At the start of the project, the regions agreed that they wanted increased transparency for public accountability. An annual performance report is published and the web platform where the data are stored is public (performance.sssup.it/netval). The report includes all measures for the regions and their HAs.

Method

Regional priorities are identified by jointly evaluating three different issues that should be relevant to them: performance; capacity for improvement; and reduction of geographical disparities. Geographical differences are important in terms of equal access: when variations that cannot be explained by population needs are found, there clearly is a problem with fairness (Wennberg et al., 2003; Barsanti and Nuti, 2014; Nuti and Vainieri, 2014).

For each indicator in the PES, the method evaluates:

• Regional performance, measured by benchmarking against the other regions.

• Trend, measured in comparison with the other regions’ trends.

• Internal variability, i.e. the measurement of the differences between local health authorities (LHAs) in the region.

The algorithm first selects, from a total of 200 indicators, the indicators whose performance, trend and variability are universally available. It then excludes the indicators that regions consider to be ‘observational’ (indicators with questionable accuracy or that are not relevant by themselves but are important to be able to understand the dynamics of other indicators). This process assures that the remaining indicators are sufficiently relevant and accurate.

Second, the procedure excludes the indicators that depict the epidemiological status of the regions (for example infant mortality or suicides): these indicators portray the healthcare need of each region. Each LHA and every regional healthcare system aims at their improvement; however, these are measured with a two-year delay and the phenomena they monitor cannot be modified in the short term by the regional healthcare governance on its own.

Once these exclusions are made, about 85 indicators (with small fluctuations among regions) are left: these indicators provide the quantitative basis for detecting regional priorities. In order to do this, the regional performance of each indicator, its trend and its geographical (intra-regional) variability are computed as follows.

First, each indicator is evaluated against the performance of the other regions or— when available—against a national or an international standard. Each region is therefore assigned a discrete score for each indicator, ranging from ‘1′ (‘poor performance’) to ‘5′ (‘excellent performance’). The score is assigned to a colour: red for ‘1′, orange for ‘2′, then yellow (score equal to ‘3′), green (‘4′) and dark green (‘5′). The same colours are associated with the trend and the variability score, described below. Colour-coding is very effective because it immediately depicts how the indicators are performing.

Second, the algorithm identifies regional priorities by considering the annual trend that each indicator shows. The annual trend of each indicator is computed for each region and compared to the trend of all the other regions. More precisely, the difference between each regional indicator trend and the median regional trend is computed; this difference is then divided by the inter-regional standard deviation of the trends, according to the following formula:

The resulting coefficient measures how far each regional indicator’s trend deviates from the network median. Each indicator is therefore assigned a second score for each region (the ‘trend score’), ranging again from ‘1′ (‘strong worsening’) to ‘5′ (‘strong improvement’). This score reflects each regional trend compared to the trends of the other regions: see table 1. An example is the changes in the percentage of femur fractures operated on within two days of being admitted to hospital from 2012 to 2013 in the different regions in the network (see figure 1). The percentage increased by nearly 26% in the autonomous province of Trento, while it did not change very much in Tuscany; therefore, considering the median and the standard deviation, the autonomous province of Trento is given a score equal to ‘5′ and Tuscany equal to ‘2′.

Figure 1. Femur fractures operated on within two days after hospital admission: 2012–2013 trend.

Table 1. Trend score.

Geographical performance variability among LHAs (within each region) is the third criterion the algorithm uses to identify regional criticalities. The inter-LHAs coefficients of variation are computed for each region and each indicator. The quintile distribution is subsequently calculated for each region. Equally, the coefficients of variation are not included for those indicators that refer to only one LHA in the region. A ‘variability score’ that ranges from 1 to 5 is attributed to each indicator, according to the quintile occupied by the indicator (see table 2).

Table 2. Variability score.

As an example, the coefficients of variation of Tuscany can be computed for 88 indicators in 2013. A variability score, based on quintile, assigns 5 to the first quintile, 4 to the second and so on: see figure 2.

Figure 2. Marche: coefficients of variation in 2013.

Finally, three different scores are available for each indicator and for each region: performance, trend and variability scores, each of them ranging from 1 to 5. It is therefore possible to sort the indicators for each region: first based on the performance score, second on the trend score, and finally, on the variability score.

Next, we added a fourth column to give policy-makers information about the financial impact each indicator might have in the short and medium term. The fourth column had a simple X next to the indicators where an improvement would result in financial savings within a year. The fourth element was added to the methodology to provide information on the relevance each indicator has in terms of resource re-allocation strategies (Nuti et al., 2010). See table 3 for an example.

Table 3. Example of the final output for region α.

Use of the methodology

Table 3 provides a way of detecting regional problems, by jointly considering the performance of each region (benchmarked against the other regions), the trend of each indicator and intra-regional variability. Each region should start its analysis by examining the first column—benchmarked performance. In table 3, a region should first take into consideration indicators C11a.3.1 and C18.6, which show a very poor performance, and subsequently move to indicators with a poor performance. An indicator requires a lot of attention if low performances are associated with worsening trends. The inclusion of the ‘variability score’ provides policy-makers with the information they need to design strategies that are tailored to single issues. If the third column warns of a strong intra-regional variability, each LHA should be assigned a specific target, based on the LHA’s previous performance. Under-performing LHAs should be required to show more improvement. However, if internal variability is low, the problem applies to the region as a whole and a structural answer should be designed, involving all of the actors of the healthcare system with uniform targets for providers.

Our technique has triggered discussions between regional managers on the priorities to be considered when setting HAs’ targets. In some cases, the tool has been used to link financial rewards, while in others it has been used to define the HA managers’ ‘mandated goals’, i.e. the results that chief executives are expected to meet in a three-year period.

The tool for setting priorities relies on information from the network’s performance evaluation system and it does not consider regional peculiarities. Therefore, it is up to regional policy-makers to decide how to weigh the three aspects (performance, trend and variability) in the priority-setting process and how to integrate the regional governance mechanisms with the evidence provided by the evaluation system.

Discussion and conclusions

This paper provides policy-makers and managers with a new method for selecting targets. The method overcomes some of the adverse effects found in large performance measurement systems. Regarding the synecdoche effect, our selection of indicators (85 were chosen out of 200) assured accuracy and relevance (accuracy and relevance are preconditions of the synecdoche principle). Second, we understood that neither the adoption of a limited set of indicators, nor the use of a wide range of measures automatically leads to distortion-free results (Holmstrom and Milgrom, 1991). However, our algorithm helps policy-makers select an appropriate number of indicators, without neglecting any important aspects of healthcare and—at the same time—without confusing anyone about priorities.

Our procedure systematically analyses all the available indicators, including all the indicators monitored by the PES in the criticalities detection process. However, the selection of the problematic indicators gives policy-makers the opportunity to set up an incentive system on a limited number of indicators, which keeps the controlled actors focused on the priorities. The mechanism we describe helps administrators set targets on ‘a part’ of the indicators, selected because of their relevance to the ‘whole’ system. Therefore, it solves the problem of using an excessive number of indicators.

Regarding output distortions, the iterative nature of our procedure—the selection of the priorities is repeated each year—limits opportunistic behaviours. Professionals cannot neglect the indicators that are not incentivized (‘suboptimization’). It detects worsening indicators (or their increase in variability) and these are automatically selected as priorities in the following year.

Bevan and Hood suggest a solution for limiting opportunistic behaviours by introducing some uncertainty in target specification. Their thesis is that excessive predictability could undermine the control itself, by facilitating opportunistic reactions (Bevan and Hood, 2004). The methodology we describe reverses their hypothesis: opportunistic behaviours are limited not by the inclusion of a degree of uncertainty but by the certainty that they will be detected the following year, when the algorithm includes all the indicators and not only the incentivized ones. The algorithm is designed to monitor a large number of indicators—and to account for the different viewpoints and goals stakeholders might have—but at the same time, it focuses the attention of the managers and the policy-makers on the indicators that are more critical in terms of performance, trends and variability.

To prevent the performance paradox, Meyer and Gupta (1994) recommend that organizations adopt a ‘paradoxical model of performance assessment’ with multiple, uncorrelated, varying, but comparable, PIs. The method we describe does exactly that. This method is particularly appropriate for use at a regional (or macro) level. In transferring this approach to a micro-level (for instance hospitals or health authorities) internal differences must be taken into account. However, if the micro-level is responsible for a large territory or a large number of providers, as in the Italian regions, this approach could be used at the micro-level. In this case, further analyses are needed to better understand the typology of indicators to be included, the effectiveness of the use of this approach at micro-level and its results.

The method presented in this paper was developed as a governance tool for regional healthcare systems. It is the first step to follow in setting targets: identifying priorities. Other techniques can be useful to help managers define what is challenging and how to conduct a fair evaluation to motivate employees (see Vainieri et al., 2016). We believe that a similar approach could be applied also in other parts of the public sector that are managed by strong professional bureaucracies (Mintzberg, 1983; Mintzberg et al., 2003), and have information asymmetry and high complexity. We think that the method would be particularly useful in the education sector (Micheli and Kennerley, 2005; Bevan and Fasolo, 2013; Bevan and Wilson, 2013; Taylor and Taylor, 2015). Further research is needed to evaluate how our methodology should be fine-tuned to be used in other sectors.

Additional information

Notes on contributors

Sabina Nuti

Sabina Nuti is a Full Professor of Management, Laboratorio Management e Sanità, Institute of Management, Scuola Superiore Sant’Anna of Pisa, Italy.

Milena Vainieri

Milena Vainieri is an Associate Professor in Management, Laboratorio Management e Sanità, Institute of Management, Scuola Superiore Sant’Anna of Pisa, Italy.

Federico Vola

Federico Vola is a Research Fellow, Laboratorio Management e Sanità, Institute of Management, Scuola Superiore Sant’Anna of Pisa, Italy.