Lost in comparison: a program theory analysis of performance measurement in social work

ABSTRACT

Since the 1990s, social work has been subject to requirements for monitoring and quality assurance. Here we examine one of the monitoring systems: Open Comparisons (OC). Gradually expanded and published online by a national agency, the OC now has around 350 indicators that cover major areas within social work in Sweden. We use program theory to clarify the operational idea in which OC is based. To do this, we analyse domestic violence data gathered from two social service organizations, from the regional level and from governmental agencies. The results show a strong normative support for OC within national level. However, OC is time consuming, its data are questionable, and its reliance on name-and-shame seems dubious. OC represents a trend in social work that may influence changes in routines and provide new kinds of performance measurements that affect how social work is organized.

KEYWORDS:

• Audit
• domestic violence
• performance measurement
• comparison
• social work

The use of instruments to measure and compare actions is a dominant trend in welfare states. To support efficiency, transparency, and equity, government organizations monitor and measure effects and work processes. Hence, these instruments need indicators that provide information on the quality of actions being performed (Pollitt 2013; Van Dooren, De Caluwe, and Lonti 2012). Such indicators are based on data from sources such as official statistics in annual reports or from targeted surveys. These reports and surveys may be produced by researchers, skilled administrators, and more recently social workers. Indicators (yardstick comparisons) were introduced around 1830 by Jeremy Bentham as a corrective mechanism to optimize production (Clarkson 2010). In the 1950s, indicators started to be used to measure welfare. During the last 20 years, indicators have gained renewed significance as part of New Public Management (NPM) and as part of the audit society in general, influencing how public administration operates. In modern strategies of governance, the use of indicators falls under concepts such as performance measurement, outcomes-based management, and results-based management.

The concept Open Comparisons (OC) is one such monitoring activity used in Sweden. OC consists of a collection of indicators developed in collaboration with state agencies and the association of local authorities (SALAR; SKL in Swedish). In 2006, health care began to use OC, soon followed by social work. To support national coordination and efficiency, the Swedish government has decided to strengthen the quality of the work performed through extensive monitoring (SOU 2015). Gradually expanded, OC now includes around 350 indicators that cover several areas within social work such as addiction, elderly care, homelessness, and domestic violence. Swedish municipal social services annually respond to surveys about their activities sent to them by the National Board of Health and Welfare (NBHW; Socialstyrelsen), a national agency under the Ministry of Health and Social Affairs (Socialdepartementet). The data are then compiled and published online. Using OC, a municipality can compare its performance with other municipalities. In addition, OC is intended to improve governmental transparency, monitoring, and support development, although international research indicates that there is limited knowledge about the effectiveness of OC (Pollitt 2013; Power 1999). Several similar indicator systems exist in Europe and the USA (Balle Hansen 2016; Pollitt and Bouckaert 2011).

This article aims to explore what happens when this monitoring is applied in social work, specifically in the field of domestic violence. Domestic violence is a complex social problem with many professional actors having partly interconnected and partly contradictory understandings of its problems and solutions. The measurement of dynamic issues is a significant challenge. If data are not compiled properly, organizations might act on inadequate information. In recent years, research on performance measurement has grown and there is a good general idea of its opportunities and challenges. In particular, research has shown that gaming (bending the rules) and cheating (breaking the rules) are inevitable features of monitoring and evaluation strategies, so further analyses of the mechanisms and contexts of these strategies is needed (Pollitt 2013, 359).

Using program theory, this article analyses how OC is applied in social work and we examine and problematize underlying assumptions in OC. We assume that the norms codified in writing as goals and agreements reveal only part of the picture. The ideal rational legal form of bureaucracy does not fit with the way organizations really act. Organizations depend on prominent ideas within their environments and catch up with organizational trends that propose how an organization is supposed to be. This approach is openly presented as a prescriptive, formal theory of change that guides organizations’ actions – i.e. what ought to be done and why. Beyond these apparent activities, we might find invisible structures, ambiguous goals, and relations of dependence. These unacknowledged aspects may be deeply embedded in everyday activities and represent an organization’s factual, informal theory of change, reflecting the way things are done rather than the way things should be done (Chen 1990; DiMaggio and Powell 1983; Donaldson 2007; Perrow 1986). This conflict between what is done and what should be done may exist as multiple program theories among stakeholders. These ideas might be explicit as well as implicit, requiring evaluators and researchers to tease out meaningful understandings (Balle Hansen and Vedung 2010). Three research questions will be addressed:

Q1. What is the formal theory of change in OC?

Q2. What is the informal theory of change in OC?

Q3. How does practice in social work handle OC?

The article is derived from a study about the design of interventions that address domestic violence. The research examines how different forms of governance – i.e. non-profit organizations, government agencies, and local social work – affect interventions using evidence and NPM (Agevall Gross, Denvall, Kjellgren and Skillmark. 2015; Skillmark, Agevall Gross, Kjellgren and Denvall 2019). Here we will offer a critical view of performance measurement and question the relevance of OC in social work. Based on policy documents and interviews, we present the overall visions and program theory of OC. We report how actors at the national level (the National Board of Health and Welfare and SALAR), regional level (County Administrative Board), and local level (two municipalities in southern Sweden) use OC in an attempt to improve the way crime victims are treated. Because no clients have been contacted, there was no need to apply for ethical approval.

The indicator movement and performance measurement

At least three intertwined factors are encouraging the use of OC: the quest for social indicators, the NPM, and the audit society. In the postwar period, international organizations such as the UN and the OECD designed specific indicators to measure welfare as a way to complement the more economic dimensions of GDP. Components such as health, education, security, and the environment were supposed to support socially-oriented urban planning. Indicators, especially in the EU, now serve as measurements of sustainability, poverty, and well-being. Indicators are considered necessary as they make it possible to overview and concisely summarize complex multiple welfare problems (Socialstyrelsen 2010, 131). Indicators may provide essential information about the social conditions in a community and the impact of social policy. OC represents a major investment in the use of indicators although they no longer measure the living conditions. That is, OC measures the quality of actions or what outcomes these actions produce. Today, they resemble the early Bentham idea of a dipstick to check employees more than the later UN idea of welfare measurement.

NPM and Total Quality Management (TQM) have received broad political and administrative support and encourage results-based public service (Arnaboldi, Lapsley, and Steccolini 2015; Balle Hansen 2016; De Bruijn 2007; Evetts 2009; Pollitt 2006). Like other organizations, social work is subject to extensive quality assurance systems and various evaluation instruments. Originally, quality assurance efforts were started by the private sector and have been described as a reaction to primitive, rationally-based goal performance. Instead, soft values were highlighted and the emphasis was on receiving feedback from employees and documenting processes (Wandersman et al. 2000). Today, quality assurance is connected with NPM and the new control system that changes the way work is managed (Hasselbladh, Bejerot, and Gustafsson 2008).

There is a wide-spread belief that it is beneficial to monitor results and processes. In the audit society, organizations invest considerable resources to measure their activities and to obtain data to visualize processes (Power 1999). This commitment to auditing has significant consequences as audit systems require organizations to design their internal structures for auditing, a requirement that inevitably depends on experts. Audit systems require continuous analysis of new data if the goal is to improve and optimize outcomes. Therefore, such information should be of good quality and give managers tools to improve governance. Critics claim too much focus is put on efficiency, cost reduction, quantitative performance, and governance models as this limited focus de-emphasizes the complexity of social problems (Arnaboldi, Lapsley, and Steccolini 2015), yet this ‘governing by numbers’ has been suggested as a new way of governance where audit technology and rating systems are core mechanisms of change (Shore and Wright 2015). Acting as a ‘performance – enhancing drug’ management by numbers might support as well as weaken performance (Hood 2012). Regardless, such ideas might serve as mechanisms of hope when searching for the rational organization (Brunsson 2006).

Indicators and social welfare

The quest for quantification and results supports the growth of metrics such as indicators, statistics, organizing ratings, and ranking systems. They support national goals of performance that may be based on data from national agencies, often statistics, presented as key performance indicators (kpi) (Balle Hansen 2016). However, it is challenging to build skills that implement and support performance measurement as extensive organizational investments are required (Hunter and Nielsen 2013; Pollitt 2013). Different logics for executing activities in organizations compete for space, which affects the quality of the data used (Høybye-Mortensen 2016). Poor input or lack of input may even create a false basis for prioritization (De Bruijn 2007; Pollitt and Bouckaert 2011; Groth Andersen and Denvall 2017). Other problems arise: synecdoche (one indicator stands for the whole performance); threshold effects (motivation below target, de-motivation, performing above target); ratchet effects (next year’s targets are based on last year’s performance); definitional drift (bending the categories to favour the organization); escalation (in time measured activities will be favoured and grow); performance paradox (the longer a set of indicators is continued, the more skilful gaming and cheating occurs); symbolic use (information is gathered for show); perverse effects (undermines morale and behaviour among staff); and discipline effects on employees (steering and correction) (Arnaboldi, Lapsley, and Steccolini 2015; Clarkson 2010, 176; Hood 2012; Pollitt 2013, 352–354; Sauder and Espeland 2009).

The more complex the information system, the more difficult it is to produce reliable data (Høybye-Mortensen 2015; Shaw et al. 2009). Transparency and information are expected to help users and patients make decisions about the services they wish to receive. However, aspects that are easily quantified receive attention, whereas aspects that are not easily quantified (e.g. care work, prevention, and long-term outcome) may be ignored (Lindgren 2014). Researchers as well as professionals themselves, especially medical professionals, have criticized the over-emphasis on quantifying performance as this type of analysis tends to ignore the role of professional competence (Hasselbladh, Bejerot, and Gustafsson 2008). The constitutive consequences seem to be significant; organizations prepare to be assessed and measured as auditees which may affect them as providers of crucial services (Dahler-Larsen 2011; Power 1999; Shore and Wright 2015).

Open comparisons

The indicators in OC are divided into outcome indicators (what the actions have resulted in), process indicators (what is actually done), and structural indicators (the conditions of the processes and results). These dimensions were inspired by Donabedian’s influential article on quality assurance in medical care (Berwick and Fox 2016; Donabedian 1966) and were suggested to be used in social work by a commission in 2007 (Socialstyrelsen 2007). Since then, indicators have been developed and in 2015 a national strategy was established. OC is defined (Socialdepartementet 2015) as follows:

[A] set of indicators that are reported annually and that reflect the activities of quality, performance and efficiency. OC is a part of the national knowledge management. (Socialdepartementet 2015, 5)

Moreover, the indicators in OC ‘should be supported by research or, in its absence, proven experience. An indicator will also give direction to a desired result’ (Socialstyrelsen 2014, 12). Six quality areas are common in the indicators presented within social services and health services: knowledge-based, secure, personalized, efficient, equitable, and accessible. Since 2011, social services units within the 290 Swedish municipalities have been asked to respond to annual surveys concerning domestic violence. In February 2016, they responded to 98 questions (about 110 pages) plus follow-up questions (11 about violence), in total a response rate of 91%. The results from the survey are presented as 350 indicators on a website where municipalities are ranked and their results are reported in tables coloured green (25% good performance), yellow (50% average performance), and red (25% bad performance). In the area of domestic violence, OC has been published on five occasions, between 2012 and 2016, and 62 indicators are expected to constitute the tools to monitor, analyse, and develop social service activities with the support to victims of violence at the local, regional, and national level. Many of the questions focus on whether the municipality has written management-established procedures (routines) for monitoring or action. ‘Routine’ refers to a specific approach for the activities to be carried out (Socialstyrelsen 2016).

Indicators are supposed to support the establishment of evidence-based practice in social work and to stimulate the development of good quality. Several of the indicators included in the OC are based on methods presumed to benefit clients/patients and where government agencies want to know the extent to which they are used. Government funding to develop the quality of social services and non-profit organizations working with domestic violence is now allocated according to whether these organizations work with methods based on scientific evidence and their responses from OC.

Indicators of OC have been researched as a monitoring technology in a limited way both nationally and internationally. This research mainly highlights comparisons within the health care system (Blomgren and Waks 2011) or performance indicators (Bevan and Hood 2006; Clarkson 2010). According to Bevan and Hood, OC focuses on what can be measured rather than on what ought to be measured. This problem stems from the fact that indicators understate the complexity of organizations with multiple stakeholders and goals (Arnaboldi, Lapsley, and Steccolini 2015). Swedish research in elderly care has shown that the use of OC is time-consuming and that the ‘name-and-shame mechanism’ might increase compliance and thereby support management control (Lindgren, Ottosson, and Salas 2012, 47; Hanberger and Lindgren 2018). No unit wants to be ashamed of its performance (i.e. be shamed into improving). The unwillingness to be associated with poor performance seems to be a central mechanism in its potential as an instrument of change, so it contributes to a ‘new mindset’ in health care. Research with a focus on social services for the elderly shows that the instrument tends to be most useful for managers and those responsible for quality issues, whereas users and even the profession’s view of quality fades into the background (Johansson and Lindgren 2013). To be effective, using OC to determine program effectiveness requires clear goals, improved data quality, and enhanced cooperation between actors; if not, what is measured gets done and what is not measured gets ignored (Vårdanalys 2012).

Material and analyses

This study derives its data from a multiple case study of two municipalities working with domestic violence with specialized units of social workers responsible for the support of abused women and their children. These municipalities had distinguished themselves for being driving forces in the development of social work for abused women, yet they were organized in different ways and used different instruments. Like most other municipalities, they participated in the annual data collection about OC. We have reconstructed the program theory of OC and related it to practice by interviewing key stakeholders and studying relevant documents.

We conducted 16 interviews with 19 participants (Table 1).

Table 1. Interviewees

Municipality 1	Top official Manager – administrator Manager – social worker Top politician Three social workers (group)
Municipality 2	Top official Manager – administrator Social worker Top politician
Regional level	Manager Manager – administrator 1 Manager – administrator 2
SALAR	Manager
National Board of Health and Welfare	Leading commissioner Manager – researcher Manager Manager
N	19

The interviewees were selected based on their different roles and perspectives in relation to the handling of OC. The interviewees are both top officials at various levels in the municipalities and regional support agencies – i.e. managers, social welfare committee chairman, and various frontline staff, including specialized functions such as women’s coordinator and development manager. At the national level, we interviewed representatives from SALAR and the National Board. The interviewees addressed their professional or political role in their work with domestic violence and OC. Interviews were recorded and transcribed by the authors. Documents were collected and analysed from the national and regional agencies as well as the municipalities.

Our analytical tool, program theory, focuses on ‘ … what must be done to achieve the desired goals, what other important impacts may also be anticipated and how these goals and impacts would be generated’ (Chen 1990, 43). Program theory tells us to look for the structures of the idea and its logical starting points. Then the researcher should trace its actual use and implementation. By reconstructing and analysing the theory of change within the program, we have targeted underlying assumptions and core logics of OC (Rogers and Weiss 2007; Vedung 2011; Weiss 1997). We have worked with three distinct analytical concepts: (A) the situation and the way problems are defined (the theory of problems); (B) the normative elements (ideas about how to conduct OC in terms of investment in personal, methods, and desired outcome); and (C) causality (why things are supposed to happen the way they do) (Chen 1990; Donaldson 2007). These concepts will be used when we present our findings. In our interviews, we asked the respondents to explain OC, how they responded to the national surveys, their views of the advantages as well as challenges of OC, and how they regarded the outcomes of OC.

Themes relevant in the analysis included the following areas: Who are the key players? What do they want to achieve? Why and how do they achieve these goals? How can the emergence and spread of indicators be explained? Data from interviews and official documents illustrate how the idea has been understood, received, and used by stakeholders (politicians, managers, and social workers) who are a part of treating domestic violence. The analysis is built on a multi-vocal and pluralistic program theory in which different stakeholders’ perceptions are considered (see Balle Hansen and Vedung 2010).

Results: the program theories of OC

Following is the analytical concepts A-C where we move from the official accounts and the factual theory to the informal theories of change underlying OC. A summary where the three research questions are answered concludes the results part.

The theory of problems

When establishing indicators, professional representatives and researchers took part in working groups initiated by NBHW. Our respondents found it difficult to develop indicators for domestic violence. Many questions and indicators were considered; ideally there should not be too many and they should assess quality. Stakeholders had different motives. Below is a quotation from a representative from SALAR taking part in the development of OC.

I think we captured many good and important areas but it is hard to ask questions so they really show the actual quality. I think that one could use it very pragmatically. That you look at the results you have and get, and use the results for development purposes. […] not to name and shame any municipalities, that’s not the point, but maybe you look at the different results in the county and compare with the nation and that sort of thing.

Which problems should these indicators deal with then? They are not easily discovered in documents from the national level. Instead, it is mentioned in general terms that better knowledge is needed about activities in social welfare and their outcomes. OC in health care was presented as an ideal by the investigating commission (Socialstyrelsen 2007, 6). Early on, this sector was subject to the new management ideals of NPM that were imported and translated from the business sector to create greater efficiency (Erlingsdottir 1995). When planning the period of 2014–2015, the problems in health care were described as the growing population of elderly, new diseases, and concentration on health centres instead of treatment at hospitals (Regeringskansliet 2014). No such description exists within social services, but later OC was described as an important tool for the implementation of improvements (Socialdepartementet 2015). Unlike health care, social services lacked sufficient registries and structures for documentation. These tools needed to be processed and national agencies have played an active role in creating a national information structure in social work (Socialstyrelsen 2007).

How then is this regarded at the local level? Opinions differ among the respondents. Several declare that they believe in OC and that it can serve as a ‘wake-up call’ and give the opportunity to compare their actions with others. One director points out that it is an instrument that makes it possible to measure their practice and outcome: ‘You get confirmation that there needs to be a change’.

But it can also be a means to draw attention to shortcomings, perhaps as an aid in the search for resources:

So I hope that there are questions about initiating investigations and things like that. It is the requirement to open an investigation even in case of an adult. (social worker)

This social worker wants support so her management can make it possible to spend more time on needs assessment for all adults who need support. Now they need to prioritize those victims who need protected housing. Other motives may be follow-ups of regulations and support to create organizational structures. They might increase the security of the individual and support the information collected. Indicators that are assessed centrally are important and favour documentation:

A high level of assessment is important for cooperation and for the work to be professional and raise quality. (manager in a municipality)

OC also faces criticism:

I think they are too focused on what documents they [NBHW] have and if they have followed up last year and so on […] you put more effort into answering the survey than you then do analysing the results […] we’ll find a scapegoat all the time. (manager in a municipality)

Other voices mention that the OC can strengthen the profession, that it will be a welcome aid in the evaluation of its actions, and that measurement is necessary. The problems the OC might solve emerge indirectly in the respondents’ stories: weak professional social workers with limited knowledge of the actions need better evaluation, documentation, and measurements to demonstrate the needs of their clients and where OC might provide support.

To sum up, not much is said in documents or interviews about the problems OC are supposed to solve. Instead it seems as if OC can be a promising audit instrument that still needs to be used and developed. OC offers national agencies more effective governance, transparency, and overview. Locally hopes are raised that indicators could strengthen documentation and argumentation about resources. Thus OC appears as a political instrument that provides a national agency the opportunity to influence social work and social workers the opportunity to argue for expanded resources.

The normative elements

Although the requirements in recent years have been formalized, normative government of social services is increasing. Together with the design of OC, the National Board has also produced several documents to support the interpretation of what constitutes quality in victim support work. Manuals, general advice, and various forms of evidence and supporting documents have been distributed. The indicators demonstrate the desirable quality of activities to support evaluation and improvement. It is also supposed to ‘measure and monitor quality in order to clarify the quality of the various stakeholders’ (Socialstyrelsen 2007, 8).

A lot of information needs to be gathered that can be measured and monitored and thus used for comparisons, both in relation to earlier documentation and in relation to other activities, locally, regionally, or nationally. The results of the quality assessment should then be visible to stakeholders, including managers, politicians, users, and citizens who are supposed to be able to have access to ‘good information about the activities and the results’ (Socialstyrelsen 2007). The National Board further emphasizes the importance of transparency and national comparisons and highlights the opportunities for follow-up and development. It seems that the social service user indirectly benefits.

How then should the OC be implemented and what are the results so far? The interviewed politicians have vague statements about OC.

I: OC, is it something you know about? Is it in use?

P: I should know, but I can’t answer that. But I can find out. (political chairman)

One of the managers in charge said that unmerited focus on documentation not only provides the board information that politicians exploit somewhat arbitrarily and the media report often sensationally but also stresses the municipalities’ resources.

The respondents find it challenging to gather the required materials. Moreover, they believe the information gathered is interpreted personally. Everyone responded based on his or her specific work, so the responses were not originally planned:

So, no, it’s delicate with OC, it’s really hard to get the issues so they can’t be misunderstood or can’t be interpreted in more ways than one. […] There are ten or eleven of us [responding to the survey], but views and opinions are tossed around in the air and somebody sits and writing at the computer. (social worker who participated in the national team)

Municipalities need to organize the answers, many people need to be involved, and it is not always the right person contributing data according to their field of expertise. One of the managers is quite critical about how they had to answer ‘No’ (marked in red on the webpage) to a question about routines even though they had an established system implemented along with a researcher they were satisfied with: ‘The response was rated red since it was not included in what the National Board considered OK’. This quote shows how the national agency tries to encourage the use of approved methods through OC.

Neither of the two municipalities uses the material regularly: ‘We take a brief look but it is of no importance’. There are many other tools and comparisons that are more specific and useful locally. Although some managers have hopes for better control and support resources, some professionals are equally frustrated that OC is not being used consistently:

We want it to be green, we don’t want it to be red [laughs]. […] I would like to see an analysis on the management level of this, open comparison is valid not just here in domestic violence without regard to the other areas in social services, we should take a comprehensive approach when looking through it in the analysis, but it has not been done. (social worker)

When the indicators hit the local level, the top-down control seems obvious:

There are very many indicators, and that they are to a large extent of a control character […] this stigma of shame should be so strong, you should avoid bad results, and that is the effect of this control. (manager at a regional council who supports municipalities in analysing OC)

According to a representative from the national agency, both control and development were goals:

But it is clear that if you look at what is the purpose of public performance of the government, it was both control and development. I think that the municipalities also felt that it was an inspection. Sharp result indicators as in health care were something to long for. Unequal social services look different in different municipalities. And you want everyone to be above a certain minimum.

Summing up: Rankings in OC trigger municipalities to organize new task assignments, and to compare their performance with other municipalities.

C. The causality

Causality is not expressed. OC was assumed to be an innovative mechanism for development but without scientific evidence of proof. Instead, OC rewards fast data for publication and comparisons to create a sense of ‘name and shame’. The latter reflection comes from one of the prime researchers at the National Board of Health and Welfare a couple of years after the introduction of OC.

In an instructive film on OC, the National Board of Health and Welfare explains how to use the tool: ‘It’s very positive and indicates the location to get together, to work together. It will be a win-win for both users, patients, and for the principals’ (author’s translation). Stakeholders are seen as consumers and are assumed to gain from increased quality of operations, so they are cast as passive rather than active actors.

What cause-and-effect phenomena do respondents identify in the municipalities? OC is not used locally in the monitoring, there are internal doubts as to how it should be interpreted, and its validity is perceived as low. To be useful, OC needs to be included in the discussions between professionals and managers. These are the experiences of a manager at the regional level who tries to support opportunities for meetings through the exchange of experiences in networking, research, and dialogue in the form of thematic conferences both when completing questionnaires and when analysing results.

Two respondents from a regional administrative board who implement the government’s action plan against domestic violence describe the OC as ‘unpredictable’ and ‘totally insane’, so they do not use OC:

The numbers they come up with say nothing, a disaster, because that depends on who they have sent it to. The municipalities that work the least were those who had the highest best results […] so you cross a bit because you do not know and you think you do not know what it means either. (manager 1)

In particular, the criteria the questions build on are difficult to interpret, and they require proper preparatory work:

It is not quite easy to compare within our area because there are always different criteria compared to […] very crazy criteria too, difficult, you can interpret them in many different ways. (manager 2)

The official ranking of municipalities using the colours green, yellow, and red could be sensationalized by the media, resulting in a lack of focus on the most important aspects of analysis and discussion:

So at the national level they have enormous expectations that the supposed stigma of shame should be so strong, you don’t want to have bad results, this is steering. (manager at the regional level)

Other stakeholders see the use of OC as a ‘flagship’ for the national agency:

[The national agency] seems to be very proud of OC; it looks great. And it should be interesting to really know if it is that good. […] I felt quite critical […] there should be other ways; it costs an awful lot of money this whole thing. (manager at the national level)

Concluding: No casual chains show the enhancement of combating domestic violence. This study corroborates other research on governing by numbers that demonstrate threshold effects, definitional drift and data distortion.

Summary

Q1: The formal theory of change.

We perceive the overall chain of governance presented at the national level as follows:

(1) Problems with domestic violence exist in society; (2) The collection of information about the problem through OC is possible; (3) OC provides an improved basis for decisions and improvement of quality; (4) Changes in routines and in methods are implemented; (5) Enhanced support for women will be the outcome. This prescriptive program theory resulted in the following flow chart (Figure1).

Figure 1. The formal program theory of OC at the national level

According to the formal program-theory of OC, indicators address the lack of overview, weak structures, and absence of impacts in social work. Local organizations will compare themselves with other organizations in the hope that this competition will enhance continuous improvement. The use of open ranking should improve the quality of the support offered to abused women.

Q2. The factual theory of change

Our analysis of the data gathered about the program theory for OC in domestic violence problematizes the official picture. The audit society and NPM provide new opportunities for governance. In complex societies, monitoring results is a way to identify where traditional bureaucracies fail in their governance and how they can improve their performance. Specifically, governments can improve their performance by using indicators to ensure the following: (1) Obtaining a richer picture of the results and processes in social services; (2) Providing better information about the work to combat domestic violence; (3) Establishing a ranking system so poorly performing municipalities can identify where they need to improve; (4) Gathering information that policy-makers can use to evaluate what procedures are in place; (5) Implementing strategies that are data driven to improve social services; and (6) Ensuring government programs are designed using standardized features (Figure2).

Figure 2. The factual program theory of OC at the national level

Open comparisons ability to improve interventions aimed for abused victims is weak. OC supports overall monitoring of social work, it provides government agencies with information and its rankings are supposed to motivate organizations to improve their performance.

Q3. How does practice in social work handle OC?

The picture of how respondents within social services handle OC varies according to position. Top-level managers and politicians support the overall idea of performance measurement, but at the same time see no immediate benefit with OC within their own organization. Middle-level managers criticize OC based on its governing effects: creating additional work, risking red markings for routines they themselves consider OK, also risking being the subject of media attention. Frontline workers perceive OC as an opportunity to get resources for the area of domestic violence and support for their professional work. The interviewed social workers are well aware that data validity in OC is unreliable. The latter critical attitude is shared by our respondents from national, regional and local level. No data supports that OC improves the quality of interventions and thus conditions for abused women.

Discussion

Since the 1990s, the ideas of new public management have caused extensive changes in the public sector. These changes have also affected how social work is performed as it has been been subject to various forms of monitoring, evaluation, and quality assurance. This research has examined one of the monitoring systems applied, Open Comparisons (OC); however, OC has been implemented without support from research. Nonetheless, strong normative support for OC exists within governmental agencies. OC provides national agencies with local data and is a tool for implementing EBP. Unlike present understandings, our analysis of the data indicates that local performance has not been affected to any significant extent by using OC even though a comprehensive administration has been created where employees of municipalities are supposed to collect data, register information, and analyse the results generated by the OC.

This study shows that the appearance of OC could be understood in terms of governance manifested in modes of control and the implementation of tools. Another driving force is the belief that indicators can be used to improve and develop social programs. The material does not discuss the supposed relation between the problems of victimized women and how domestic violence should be solved using indicators. However, the quality of performance in social work is in question. National agencies use these indicators to compare the effectiveness of social programs provided by municipalities, a strategy that is presumed to specifically improve the overall quality of the services provided to victims of domestic violence. However, no distinct casual chains are presented. Instead mechanisms are recognizable from other research on performance measurement where naming and shaming and ranking systems are introduced to support quality. The impetus for OC is to be found at the national level of public government. At the local level, social workers view OC as an added burden, at the same time as they express hope of OC as a tool for improvement.

National agencies have dual roles. On the one hand, an agency is supposed to implement and develop the use of OC. On the other hand, the data are used in other parts of the agency’s support regarding domestic violence despite the poor quality of data. Information from the OC is used to provide financial assistance to municipalities. Only when certain procedures are in place will the municipality receive some support to develop methods. Our study shows that the municipalities themselves are well aware that their answers are not always accurate.

This research has identified OC as a normative technique that enhances organizational self-regulation. In addition, OC is assumed to provide benchmarking, continuous quality improvement, the limitation of discretion, and standardized work performance. Performance measurement indicates the OC results in increased management and new methods of control. However, indicators do not stand alone; they need to be interpreted, which calls for networking to analyse the results. OC promoters seem, however, to be unable to convince stakeholders that it is an effective tool for managing information and knowledge. That is, knowledge management is realized through constitutive formation. OC, as a modern tool, influences how social work is performed by relying on competition as a mechanism of change (i.e. name and shame), and this strategy seems unavoidable. Those who do not take part are met with suspicion. Staying outside OC might be an option, but who dares as there might be consequences. Our respondents noted that the media can place undue pressure on municipalities that perform ‘poorly’ by publicizing comparisons.

OC might be a driving force for municipalities that come out poorly in the ranking, encouraging them to invest in services aimed at supporting victimized women. A ranking system could reward municipalities that perform well and punish those that perform badly, but no data support this widely-believed assumption (Hertting and Vedung 2009, 38; Sauder and Espeland 2009). Ranking systems are influential and change the preconditions for organizations that must react. Organizations are exposed and under external pressure. According to the theory, this pressure will push poorly performing municipalities to implement routines and methods using strong scientific support. That is, OC is believed to be a driving force of governance.

Indeed, OC provides national agencies with data and has succeeded according to the program theory in implementing a new routine in the social services. It is now on the local agenda. This research has shown that surveys and variations in the registration of data indicate validity is low, leaving responders with incorrect information. The data presented on the website are unreliable according to respondents in the two municipalities. OC, however, does not solve the information needs of national agencies since the data provided might be unreliable.

Social work is complex and indicators are unable to show more than partial pictures. The picture in this version of OC does not consist of outcome data or data directly from users of social services. That is, knowledge about impact is not provided. OC delivers data about routines, processes, and to some extent structural information. Donabedian’s original idea from 1966 about outcome data has been put aside; instead, OC sends messages about the importance of administration and routines.

OC requires human and financial resources. Professionals working with clients exposed to domestic violence are often already burdened with a heavy workload, but they feel a need to prioritize the surveys so their municipality will not be perceived as performing badly as competition and benchmarking are the sources of evaluation. OC does not provide insight and explanations and does not provide answers to questions about the outcome of social work. Instead of organizing proper evaluations, municipalities must put up with unreliable data gathered from the OC. Furthermore, OC concentrates on the appearance of procedures within the organization. These procedures do not guarantee that clients receive effective services. Indicators of quality can be valuable, but their usefulness requires careful professional judgements.

The limitations of this study rest in its empirical design. As this study relies on only two cases, there is a need for careful generalization; we do not know whether the results would include other areas of social work or even health care. Therefore, the analysis had to focus on theoretical generalization. The research presented here provides a snapshot and the circumstances may have changed since the study began.

Conclusion

We conclude that the results published in OC about interventions in social work in the field of domestic violence are built-in without proof of causality. Implications for the national level include improving the quality of data gathered for OC. Indeed, this will take resources, but if the overall goal is the use of data for governance, there is no other way. Otherwise, the national level might end up with endless discussions using ‘alternative facts’. At the local level, providing resources for valid and real evaluations would improve the services provided to clients. That is, municipalities are now using their limited resources to complete national surveys they do not use themselves. The original goal to use indicators to collect knowledge about people’s living conditions has now been reduced to a matter of accountability.

OC represents a trend in social work that reflects the global influence of performance measurement. It changes the modes of governance from direct governance to governance by numbers. Today, organizations demand the proof of the implemented processes and effects of their interventions. OC changes routines and provides standardized information that affect how social work is organized. Our results show that OC can be used as a political instrument for controlling social work and to argue for expanded resources and in politics, validity and correct documentation are of minor importance. Finally, does the use of OC improve the lives of women who are victims of domestic violence? These victims deserve a credible answer.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

The Swedish Crime Victim Compensation and Support Authority, grant [04049/2013].