Contextualizing the trend from output to outcome measurement: the Dutch pension system

Abstract

Measuring outcomes rather than outputs is assumed to increase transparency in reporting. The authors discuss changes in the performance indicators used in the Dutch pension system from output to outcome measures. Expectations were not met. The main outcome assessed related to the viability of the system in the long run. A side-effect was that transparency for the stakeholders was diminished.

Keywords:

• Outcome
• output
• performance measurement
• public pension systems
• transparency

Public sector performance measurement is considered to be changing from using predominantly output measures to using outcome measures (Bianchi, 2016), but what does that mean? Many dimensions are involved in distinguishing outputs and outcomes. We demonstrate this by presenting the changes in performance measures in the pension system in The Netherlands. The changes were intended to increase transparency and to provide more contextual and meaningful information about performance (De Vries, 2010; Van Dooren et al., 2015). In this paper, we look whether these expectations are reasonable.

Recently the Dutch pension system has changed significantly. The use of performance measurement of pension funds reflects these changes. The main question we address in this paper refers to the way in which these changes conform to a shift from output to outcome measurement and whether such changes do increase transparency. In order to answer these questions, we first address the different dimensions involved in distinguishing between output and outcome measurement. Next, we present the visible changes in the dominant indicators used for the measurement of the performance of the pension system in The Netherlands. The paper continues with an assessment of these changes in terms of the different dimensions of output and outcome measurement, after which the main question is answered.

Outputs versus outcomes

‘Output’ and ‘outcome’ are abstract terms and vary in their meaning according to the problem at hand. Some see both outputs and outcomes as impacts of policies and decisions and take their meaning for granted. Quoting Vedung (1997, p. 5): ‘another term for outcomes is impacts’). According to Goodykoontz (2014, p. 3): ‘Throughout this guidebook, we use the terms results, impacts and outcomes interchangeably. All three terms can be used to describe the effects of a program or activity, particularly its achievement of or progress toward established goals’. Others have explicitly distinguished outputs from outcomes and impacts. For Levy et al. (1974, p. 2) output is the action to be undertaken following a decision. The resulting resource allocation would denote the outcomes and the altered lives of citizens would be the impact. In a similar way, but resulting in a rather different distinction, Wolf (2010, p. 4) defines outputs as the (self)commitments of actors, outcomes as behavioural changes based on such commitments and impacts as a contribution to problem-solving or goal attainment resulting from behavioural change. Wolf says that outputs are verbal, outcomes imply action and impact is effect.

This very brief introduction to the subject highlights the variations in interpretation. Next, we present a more extensive overview of the variations in the meaning of the terminology in order to ascertain the different dimensions involved in distinguishing outputs from outcomes and to explain the changes in performance measurement in the Dutch pension system must be understood.

Outcomes as changes in the input to the system

The concept of outputs relate to general systems theory, of which the origins in political science can be traced to Easton (1957, 1965). Easton does not distinguish between outputs and outcomes. His model was an input–output analysis, primarily about the maintenance of the political system and its need to adapt to technological and environmental changes, as well as changing demands, in order to survive. To accomplish this, the system receives inputs (support and demands) and converts these inputs into binding decisions (outputs). The output results, through a feedback process, in changed demands (input). The outputs of the political system are also the inputs into other systems, i.e. its environment, as well as the changing input in its own system. According to Easton (1957), a major task for research is to ‘establish the relationship between outputs and succeeding inputs of the system’.

This idea was based on Von Bertalanffy's (1968) model for understanding systems in general, irrespective of the scholarly discipline. This model ‘is a circular process where part of the output is monitored back, as information on the preliminary outcome of the response, into the input, thus making the system self-regulating; be it in the sense of maintenance of certain variables or of steering toward a desired goal’ (Von Bertalanffy, 1968, p. 161). This idea was used by Olomu (2011): ‘When a new policy interacts with its environment, outcomes may generate new demands or supports and groups in support of, or against the policy, will bring new input (feedback) or a new policy to the related issues’. Implicitly, this theory perceived outcomes to be equal to changes in the input (demands and support) to the political system, because of the output it produces.

Outcomes as consequences of purposeful action

It was through functional theory, and especially the theories suggested by Parsons (1949) and Merton, that outcomes explicitly entered the model. From this perspective, ‘output’ denotes the manifest functions of a political system. It refers to purposive action, i.e. conscious action that involves motives and consequently a choice between various alternatives. Outcomes would refer to the consequences of such purposeful action those elements that would not have occurred had the action not taken place (Merton, 1976, p. 895).

This functional theory thus perceives outcomes as all consequences of purposeful action (output), irrespective whether they are intended or unintended and irrespective whether they refer to consequences for the system itself or its environment. In order to be designated as ‘outcomes’, developments have to be directly related to outputs, although they need not only refer to the achievement of the goals and objectives. There doesn't need to be a one-to-one relationship, because the effect of outputs on outcomes can vary in different contexts (Pawson and Tilley, 1997). Pawson and Tilley see outputs as mechanisms that in combination with the context result in outcomes. They point out that ‘[t]he configurations of context, mechanisms and outcomes explain what works for whom in what circumstances’, resulting in the formula ‘outcomes = mechanisms + context’.

Later researchers both extended and limited the concepts. Levy et al. (1974) asserted, for instance, that in the public sector output is seen in taxes and expenses, while outcomes refer to the benefits for the citizen. Vedung (1997) also sees output and outcomes as both being the results of a policy. Output is meant for phenomena that come out of government bodies in the form of, for example, prohibitions, enabling procedures, grant, subsidies, taxes, exhortation, ‘jawboning’/‘moral suasion’, services and goods. Outcomes reflect what happens when the outputs reach the addressees. In the same way, Forsythe (2000) claims that ‘output is the narrow measurement in numbers of tasks accomplished, while outcome is the broader perspective on the impact on the environment’. Radin (2006, p. 2) considers output as resources used, specific activities that emerged from the organization, while outcomes refer to the impact of those activities.

Outputs are system-oriented; outcomes context-oriented

In another interpretation, the link between output and outcome is loosened. Output would refer to the system delivering decisions and policies, while outcomes are mainly related to the environment of the system, be it because of the way the outputs are received, or because of other causes beyond control. Bemelmans-Videc asserts (2007, p. 243) that there is a need ‘for a realignment of accountability practices from attention mainly on process or outputs, that can (or ought to be) controlled, to much more a focus on outcomes that cannot be completely controlled’. Similarly, Ahmad and Eijaz (2011) state that, output refers to decisions, while outcomes are the development of the system in terms of decay and progress, which may or may not be consequences of those decisions. This dimension emphasizes that thinking in terms of outputs is indicative for being system-oriented, while stressing outcomes is more environment-oriented. Therefore, adherents to this view tend to loosen the criterion that outcomes need to be a consequence of outputs, as outcomes in the environment of the system can appear irrespective of output from the system.

Output as immediate effects; outcomes as intermediate and long-term effects

A fourth dimension is time frame. Output would be the immediate effect—more police on the streets, more money for education, more patients in hospital beds etc.; while outcomes relate to intermediate and long-term effects—for example safety, learning and physical wellness (Afonso et al., 2010). This is seen in a World Bank report where: ‘Country inputs include programs and development investments undertaken by countries to improve fiduciary management, initiatives to improve the quality of education, policy reforms to improve the investment climate, etc. These reforms, or country inputs, in turn lead to outputs, such as, for example, reduction of barriers to private investment, school enrollment and immunization. Ultimately, these outputs, over a period of time, lead to outcomes which are overall country measures such as gains in literacy, increased life expectancy and lower infant mortality’ (IDA, 2002, p. 1) The same distinction appears in other policy reports, for example Save the Children (2008) which defines outputs as the immediate results of project or programme activities (for example children receive training, community mechanisms are set up) and outcomes as the intermediate changes as a result of project or programme activities.

Output is what is actually done; outcomes refer to the evaluation of what is done

Others see the difference as one between descriptive and normative reasoning. As Smith (1996, p. 1) said: ‘the purpose of measuring outcome is then the valuation placed on the activity’. He presents the formula, ‘Outcome = Valuation (Output*Quality)’ (p. 2). In this sense, output addresses what government does to achieve goals; outcomes are defined in terms of the success or failure of what government does (Atkinson, 2002). This is also implicit in Melkers and Willoughby (2005, p. 183) who define outputs as the measures of the quantity of services provided—for example, the number of lane miles of road repaired or the number of serious crimes reported—and outcomes as measures of the results that occur, at least in part, because of services provided, for example, the percentage of lane miles of road maintained in excellent, good, or fair condition, or the ‘clearance rate’ of serious crimes, or the percentage of residents rating their neighbourhood as safe or very safe. Also according to Boyne (2002, p. 18), outcomes include a central criterion for the evaluation of organizational performance in the public sector: the equity or fairness of service provision (as assessed, for example, by the distribution of outputs by gender, age, race, income and geographical area).

Outcomes as micro-outputs

From a managerial point of view, output and outcome are both inherent to the system at hand. In this conception, outcomes also appear inside the system. Their line of reasoning is that you have the broad decision (the output), but in the actual implementation you need to know, in what way that broad decision works out in the norms of the service-rendering personnel and the different types of services actually received by various groups (outcomes) (Kirlin and Erie, 1972). This relates to the original idea of outputs as decisions and policies (Easton), in which the system tells what it is going to do and outcome referring to the realization of those intentions through the functioning of the organization.

Outputs as easily measured and outcomes as fuzzy concepts

A common distinction is that outputs are easily measurable, while outcomes are often fuzzy, more abstract and not easy to quantify— concepts that ‘do not easily lend themselves to quantification’ (Jacobs et al., 2013, pp. 4–5). Raw data and measures in terms of numbers, frequencies and percentages are outputs, while composite measures as in one-dimensional and multidimensional scaling and factor analysis are outcomes. As Romzek (2000, p. 34) notes, ‘When relying on outputs, there is a tendency to emphasize measures that are easily obtained … Although most reforms aspire to increase the use of outcome measures … there are substantial difficulties in developing and using performance indicators to measure results’. The search for outcome measures can be illusionary, because of the many issues at stake (see De Vries, 2010, p. 4).

Our seven dimensions distinguishing outputs from outcomes are summarized in table 1. The overview and table indicate that there are multiple ways to distinguish between the two abstract concepts. Although there is more or less agreement on the meaning of outputs, there is a huge disagreement about the meaning of outcomes.

Table 1. The seven dimensions of outputs versus outcomes.

Dutch pension provision

From the mid 1950s onward, the Dutch pension system has been based on the three pillar system (OECD, 2015, p. 312) in which government and private sector each have their own, intertwined, roles. The three pillars include a compulsory government organized basic pension plan, a quasi-compulsory— employee based—collectively but privately organized supplement and a voluntary individual life insurance annuity. Originally, the system centred on the idea of a defined benefit (DB) system based on participants’ final salaries.

Hence, the system is hybrid, which means that funding is based on a mixture of pay–go benefits and capital accrual benefits. In its most recent study, the OECD (2015, p. 312) shows that the net replacement rate of income ranges between 90 and 103%. This is due to the high levels of the compulsory second pillar savings (Allianz, 2015, p. 18; OECD 2015, p. 69). Government is responsible for the first pillar pay–go pensions, which are not means tested. Everyone who has lived in the country for at least 50 years receives the full benefit at the retirement age of 65. The main change in recent years, from 2013 onwards, is that the retirement age has been gradually adjusted to 67 and could further increase based on life-expectancy changes.

Most employees (about 90%; OECD, 2015, pp. 189, 312) participate in quasi-mandatory accrual-based second pillar plans. These plans are set up by social partners (employers and employee representatives) and run by private independent pension plans. In most cases, these plans are currently based on life-time average earnings, rather than final salary calculations (OECD, 2015, p. 313). Particularly in single company pension plans, a trend towards defined contribution (DC) schemes can be observed (Frijns Committee, 2013, p. 12; Bovenberg and Gradus, 2015).

Central government in The Netherlands has set fiscal and regulatory standards for the pension industry. This, in combination with the significance of privately-organized second pillar plans, means that the pension system in The Netherlands can be thought of as a form of co-production between government and the private sector (Alford, 2015). In line with Easton's approach on systems theory, the fiscal and regulatory outputs of government can be viewed as inputs into the second pillar plan systems. On a more individual level, government outputs in terms of first pillar benefits are part of the inputs of the individual income which is supplemented by the inputs of the private second pillar benefit.

If an individual assesses the outcomes in terms of support for the pension system as a whole, the aggregate income from both pillars should be in line with expectations or promises made within the system.

The fundamentals of the system were more or less stable until the dot-com bubble crisis in 2002. After 2002, three major changes were introduced:

• First, the traditional final salary based pension was replaced by an average life-time income based system in 2004/2005 for the majority of participants.

• Second, due to financial market developments, the traditional calculation of liabilities was transformed into a system based on market rates, rather than on a fixed rate liability calculation. This was introduced in 2007 as part of a complete revision of the regulatory framework for pension provision.

• Third, increased longevity and fiscal distress led to a longevity based increase in official retirement age from 2013 onwards.

A fourth change is under consideration and essentially focuses on a shift from a DB to a form of DC system as advised to government by employers and employees (SER, 2015). The overarching effect of all these changes is that the risks involved in the amount of pension someone can expect have been transferred to individuals. It is now much more difficult, if not impossible, for individuals to calculate their expected pensions (Frijns Committee, 2010, p. 21).

Empirical assessment

When assessing the changes in terms of outputs and outcomes, we need to be aware of the differences between participants. Current practices (early 2018) make distinctions between ‘start-career, mid-career and end-career active participants’. Furthermore, retired participants, as well as non-active participants not receiving benefits, can be identified. Given the premise that accrued pension levels cannot be affected by changes in the system, all the changes have a forward-looking impact. So the impact of changes and risk transfer is relatively large for start-career participants and relatively minor for end-career participants.

Output and outcome in the final wage DB system

In the original (pre 2007) system, the key issue on the system level between government and pension providers was the solvency of a plan, defined as the available funds divided by the liabilities discounted at a fixed 4% interest rate. Regulation required solvency levels above 100% and this was in general not an issue. In the mid 1990s, due to high funding levels, participants were given premium holidays (Frijns Committee, 2013, p. 14) and the government discussed whether some part of the high funding levels should be taxed. Essentially, after taking a more risky asset allocation from the 1990s onward, additional wealth was accrued due to high market valuations of assets. This development can be assessed from a context-oriented outcome perspective: markets, in combination with revised asset allocation by pension plans, created substantial additional wealth (Frijns Committee, 2010, p. 35).

There is also a time dimension in both output as outcome in this development. The immediate effect of premium holidays was that the net income of participants increased, because they did not have to save for their pension. This generated additional legitimacy in the short term. The long-term effect, however, was that additional accruals came from financial returns rather than from cost-covering contributions. Nowadays, these premium holidays are strongly contested, particularly by retired participants who are facing possible benefit cuts. So, the legitimacy of the intervention on the long run is not as strong as it was when the intervention was realized.

On the individual level, an active participant in the pre 2007 setting only had to know the number of years he/she had participated in the plan in order to calculate the his/her retirement income. Basically, when working full time for the same employer during one's whole career would generate simple concrete outputs and understandable long-term effects in terms of expected retirement income. For those not working full time for the same employer, the long-term outcomes were uncertain and were often only clear at retirement.

The transition to average wage accruals

A substantial change in the second pillar pension plans was implemented in 2004/2005. Employers and employees, negotiating the pension plans, took the initiative. The essential point was that a final wage DB system was becoming expensive mainly due to changing demographics. The change to an average wage DB system implied that the additional premiums needed to cover wage increases were no longer necessary. The immediate effect of the change was that annual premiums stabilized rather than increasing. The intervention intended to reduce costs in an effort to maintain support for, and delimit the demands on the system as a whole, from a system dynamics perspective. At the pension plan level, it meant that the ex ante cost to cover premiums became the key indicator in the annual decision on premiums to be levied for the next year. This seems to be a straightforward indicator, but essentially it is an abstract concept as regulation allows many underlying assumptions and indicators which can be adjusted to calculate the proposed premium. In general, only the experts supporting pension plans really know the impact of choices.

On an individual level, most participants were unaware of the change (Frijns Committee, 2013, p. 14). Individuals might observe a change in their contributions, but that was all. From the introduction of the average wage system, the expected income at retirement became a more important indicator.

To try to mitigate the effect, employers and employees agreed on a system in which investment returns could be used conditionally for compensation for inflation. This led to a new output indicator: level of compensation for inflation granted annually. The indicator has some system dynamics characteristics as it is a decision. Furthermore, it is relevant from an immediate, as well as from a long-term perspective and has a descriptive character. Retired plan members see their actual compensation for inflation on their pay slip. The accrued entitlements of other participants also increase but are not immediately visible.

At this stage, both key indicators continued to have a descriptive output character. The information on expected and actual levels of accrued annual pensions included information on compensation for inflation. Whether or not a gap existed between intended or expected pension levels was not communicated.

Given that already accrued pensions up till the systems change were not affected, the impact in terms of outcome should be assessed on a differentiated level.

For retired participants, a context-oriented assessment was needed: if no compensation for inflation us provided, their immediate purchasing power would be affected, but purchasing power could be affected by government interventions as well. For participants starting out in their careers, with hardly any previous accrued pensions, the impact is that they will receive a pension based on their average life-time income. For end-career participants, the impact is substantially lower, only the remainder up to retirement is affected and only if individual wage increases more than inflation levels. A 60-year-old participant receiving maximum level of wages at the introduction of the average wage system will face similar effects as retired participants. Such differences in impact were, however, not communicated to the public.

The 2007 regulatory framework

The original regulatory framework for pensions was created in the 1950s. Over time, the labour market changed and the sustainability of the second pillar pension system was in doubt as a result of the financial crisis and demographic changes. Both government, employer and employee organizations acknowledged this and worked jointly on a revised regulatory framework, which came into effect in 2007. This framework formalized the already existing practice on conditional inflation compensation. It also changed the system of valuation of liabilities from the previous fixed 4% system to a market based interest rate system. This change led to two new performance indicators, mainly relevant for the pension plans.

The two new performance indicators are descriptive and focus on actual market interest rates and the level of interest rate hedging as a tool to mitigate interest rate risk. To some extent these indicators are input indicators driving the financial result of the pension plan as expressed in the funding level. The funding level determines conditional inflation compensation and, in case of insufficient funding levels, interventions by the pension plan to arrive at minimum required funding level of 105% are needed. Hence, funding levels and the resulting inflation compensation became more visible to individual participants.

Although the interest rate hedge indicator is a descriptive indicator, it also has some characteristics of an outcome-effect indicator. Essentially the indicator shows the impact of a 1% change in interest rate on the funding level. Without hedging, the funding level will change with the same amount as the duration of liabilities.

In 2007, the new indicators were, in practice, only relevant from a pension plan board and regulatory perspective. In 2005/06, the interest rate was under 4% and in 2007 it was substantially over 4%. This meant that liabilities decreased compared to calculations in the previous regulatory framework. It also suggested that the new indicators were not seen as relevant for individual participants.

Longevity and fiscal distress

When global financial markets collapsed in 2008/09, one of the effects was that funding levels dropped substantially below minimum required levels. Government intervened by adjusting the time frame for recovery from three to five years. Due to the further decline in interest rates, this did not realize intended effects.

The financial crisis, as well as previous changes in the system, had highlighted the vulnerability of the system to external variables. By 2010, a new and substantial variable emerged on the public agenda. A new system of calculating longevity by independent actuaries and later on from the statistical office indicated strong increases in longevity (for example Van Ewijk et al., 2014, p. 13). Given the characteristics of a DB plan, this meant an increase in liabilities of some 8% and thus a similar drop in funding level for an average pension plan. Longevity, however, also affects the first plan pillar plan, because payouts for participants also will rise each year. This, combined with fiscal distress, led to the adjustment of retirement age from 65 to 67 years in 2021 and a fixed rule for later adjustments in the first pillar plan and adjustment of fiscal accrual regulations based on a standard retirement age of 67.

At the level of co-production between government and second pillar plans, implementation issues were created, but no new performance indicators. In terms of the relationship between second pillar plans and their participants, new indicators were generated:

• The difference between the actual and expected retirement age of participants became an indicator.

• Expected retirement income had to be adjusted on an actuarially neutral basis to the new official retirement age. So the expected income at retirement reported to active participants changed. In addition, people wanting to retire at the original 65-year-old retirement age face lower benefits compared to the officially reported income.

So, over time, the performance indicator ‘expected income at retirement’ changed from a fairly concrete final salary based figure to an abstract concept based on average life-time income at a moving target retirement age.

Although regulations on recovery plans were adjusted, some pension plans had to implement substantial benefit cuts during 2013. By mid 2015, the regulations for benefit cuts were adjusted so that cuts can be spread over a longer period of time and re-evaluated annually—this creates even more uncertainty for participants.

All the above resulted, first, in a system of which the legitimacy is under pressure; this is why a search for a simple and straightforward pension system is again on the political agenda. Second, risk is transferred to participants. For someone wanting to assess their financial position after retirement, two tools have been developed. These tools are not performance indicators but, rather, they are outcome based tools intended to change the behaviour of participants. The first tool is descriptive: in a digital and government regulated environment, all first and second pillar entitlements are disclosed to the participant, provides an idea of expected income after retirement. In addition, most second pillar plans have developed planning tools which allow to match expected retirement income to expected expenses after retirement.

Conclusions

We have analysed the various meanings of output and outcome measurement, producing seven dimensions. The definition of outcome measurement is particularly complex. We have identified seven dimensions of outcome measurement ranging from very abstract to more concrete effect-oriented approaches.

Subsequently, we used changes in the Dutch pension system, and changes in the performance measurement systems employed, as an illustration of a change from output to outcome measurement in its different dimensions. We wanted particularly to find out how much impact the changes had had on transparency. If the changes in the indicators can indeed be seen as changes from output to outcome measurement, the main outcome seems to be system-internal: namely the reduction of the increasing burdens and demands on the pension system itself. The long-term viability of the system was the main outcome we assessed. We did this to conform to Easton's idea that the outcomes as produced by the systems’ outputs are the changes in the input (demand and support) for the system itself. All the other dimensions of outcome measurement seem to be more or less secondary.

The main outcome indicator measured the demands placed on the pension system and its legitimacy, which is determined partly by contextual dynamics. As to transparency, the new performance measures did not give the stakeholders any more information. Instead, the performance indicators were important for the funds themselves and for the controlling agency, in terms of whether they were going to be able to take care of their liabilities in the long run.

Our case study has shown that a change from output to outcome measurement does not need to be indicative for an increased external orientation and does not by necessity increase transparency. Rather, the case of performance measurement in Dutch pension funds points to the opposite, namely a focus on the internal system and less transparency for the stakeholders regarding the pensions they may expect and the contributions they have to pay.

Although pension systems, as we face ageing societies, financial downturns and low interest rates in the EU could be a special case, our analysis nevertheless points to the continuing importance of the original systems theory in which real outcomes are seen as to be nothing more than the changes to system inputs.

Additional information

Notes on contributors

Johan A. M. de Kruijf

Johan A.M. de Kruijf is Assistant Professor at the Institute for Management Research, Radboud University Nijmegen, The Netherlands.

Michiel S. de Vries

Michiel S. de Vries is Full Professor and Academic Chair at the Institute for Management Research, Radboud University Nijmegen, The Netherlands.