Evaluation systems in a crowded policy space: Implications for local school governance

Abstract

Evaluation systems of various types are an integral part of a country's education policy space, within which they are supposed to have the basic functions of enhancing accountability and supporting school development. Here we argue that in a crowded policy space evaluation systems may interfere with each other in a way that can have unintended consequences and create new ‘policies by the way’ that are not the result of intentional policy decisions. To shed light on this argument, we examine five of approximately 30 evaluation systems operating in the Swedish education system. Our analysis examines a situation in which many evaluation systems are doing almost the same thing, i.e. collecting a similar and limited set of quantitative data, and addressing the same local governance actors with the primary goal of supporting school development in the same direction. By doing so, these evaluation systems could thus give rise to several unintended consequences, including a scaling down of the school law and curriculum, multiple accountability problems, increased administration and new intermediary job functions at the level of local education governance.

• evaluation systems
• crowded policy space
• policies by the way
• unintended consequences
• multiple accountability problems

Introduction

Evaluation and related practices such as auditing, inspection, and performance measurement have grown extensively worldwide and are today regarded as key tools of contemporary public sector governance (Pollitt and Bouckaert 2011). Education is one of the social sectors that is most densely populated in terms of evaluation (Lingard 2011). Evaluation is clearly not an end in itself; rather, evaluation is incorporated into a given governance structure and assumed to fulfil or support particular functions, basically to steer actions and behaviour, while guiding organisations and professionals in a certain direction.

Though not unique, Sweden is a case in point with an increasing amount of evaluation at all levels of its education systems. Most of this evaluation is conducted by public authorities, but private and civil society actors are ever more conducting the evaluation of education. A problem with the existing evaluations and a key conclusion of two recent reviews is that while certain elements of education evaluation are well established, they are not well coordinated (OECD 2011, 2015). Observers even speak of the risk of overcrowding in the arena of education evaluation, resulting in multiple accountability problems (Lindgren and Hanberger 2014).

Also not unique to Sweden is a paradigm shift in the organisation of evaluation, described as a shift “from studies to streams” (Rist and Stame 2006), and implying that stand-alone evaluations, done by experts for particular purposes at single points in time, are losing ground, giving way to mandatory, repeated and routine evaluation systems operated by organisations and institutions (Dahler-Larsen 2013; Leeuw and Furubo 2008). Comparative studies of education policy in Europe have described this shift as a metamorphosis from cultural education policy into ‘governance by numbers’ in which standards, data, indicators and benchmarks dominate education discourse and practice in many countries (Lawn and Grek 2012; Rizvi and Lingard 2010). This shift has also enabled public and private actors to use publicly available statistics to set up new evaluation systems.

In studying education evaluation systems, the focus of the analysis has typically been on individual systems or on comparing individual systems across countries, for example, systems of student assessment (Gustafsson, Cliffordson and Ericson 2014) or inspection (Ehren et al. 2013), or on international comparisons (Grek 2012). In contrast, the present article looks at the body of existing evaluation systems in one country. The article is informed by policy analysis that conceives of policies as occupying a ‘policy space’. This term was introduced by Wildawsky (1979) who contended that the consequences of one policy are likely to interfere with the working of other policies, and that the possibility of such interference increases in a crowded policy space. Accordingly, we argue that education evaluation systems are policies in their own right, i.e. solutions to perceived problems that help achieve certain objectives, that also constitute an integral and increasingly important part of a country's education policy space. Within this space, evaluation systems are likely to support education policy and governance, but may also have unintended consequences for each other and for education policies. For example, a private organisation's evaluation system operating in a crowded policy space may transform problems that are intended to be addressed by applying public evaluation systems, making this system a problem for national education policy. The evolution of new scrutinising organisations (Keane 2008) implies that national and local governments must respond to assessments from actors in a policy space.

The aim of this article is to provide an overview of existing education evaluation systems in Sweden, and to analyse, compare and discuss the possible consequences of five evaluation systems for local school governance. Space does not allow for a comprehensive discussion of the possible consequences of all identified evaluation systems (of which there are 34). For the purpose of this article, we selected five authoritative evaluations systems applied in compulsory (i.e. primary and secondary) education that, according to our own and others’ research, are considered dominant in school governance (Hanberger, Lindgren and Lundström 2016; Blanchenay, Burns and Köster 2014). The present article applies a policy space approach in analysing the coexistence and consequences of multiple evaluation systems so as to advance our understanding of evaluation in the Swedish education system. The findings presented here contribute to research on education evaluation systems and are likely to be of value to politicians and managers involved in education policymaking. As many other countries have witnessed the growth of evaluation systems in the field of global education (Lingard 2011), the findings are also likely to be of value to actors in other education systems.

In what follows, we first briefly introduce the Swedish education governance system and provide a general geography of its evaluation arena, i.e. the context in which the investigated evaluation systems operate. Next, we outline a conceptual and theoretical framework, and a methodology that will guide the examination and analysis of the five selected evaluation systems. Then, the five evaluation systems are first explored one by one, and subsequently compared with regard to their key features and intended functions. Lastly, we conclude by discussing the implications of these findings for local school governance in light of the conceptual and theoretical framework.

Swedish education policy and governance

Since around 1990, the previously comprehensive and centrally regulated Swedish education system has become a decentralised, quasi-market and goal-oriented system, with responsibility for schools having been shifted to the municipalities and the local government level (Lundahl 2005). The country's 290 municipalities are responsible for employing school staff, organising schools and adult education, and determining resource allocation; moreover, they are also accountable to the central government for their schools’ achievement of national education goals and requirement levels. In addition, local school actors are responsible for implementing the Education Act which defines an overall school curriculum divided into fundamental values and tasks of schools, overall goals and guidelines for education, and syllabi with knowledge requirements (SFS 2010:800; SFS 2011:185; Skolverket 2011).

The quasi-market Swedish education system is based on school vouchers and the right of parents and students to choose freely between publicly financed schools, owned and organised either by municipalities or independent school providers. In 2014, about 17 per cent of compulsory schools and 50 per cent of upper secondary schools were independent schools (Skolverket 2015a).

Over the last decade, student performance in compulsory school has declined dramatically, differences between students and schools have increased, and the equity level has deteriorated (OECD 2015). To address these and other problems, the decentralisation of the 1990s has given way to policies of re-regulating and re-centralising education governance (Eklöf et al. 2009; Hudson 2007). This policy shift has been described in terms of decentralised “management by objectives” being replaced with “centralised and performance-based management” (SOU 2014:5, 30). This shift has gone along with a wave of reforms, including a new Education Act with distinct learning goals and assessment criteria, reinforced and extended to more subjects, and a Schools Inspectorate with enhanced tools to sanction schools that fail to meet national requirements (SOU 2013:30).

Swedish education governance is a system with several layers. The Education Act and the government's objectives for education constitute the basis of education governance. On behalf of the Ministry of Education and Research, the National Agency for Education (NAE) formulates and proposes goals for the curricula, manages the collection, analysis and dissemination of national statistics, and conducts national monitoring and evaluation. The Swedish Schools Inspectorate (SSI) supervises all schools. Municipal and independent school providers are responsible for schools, implementing educational activities, organising and operating school services, and for self-evaluation by monitoring school quality and results to ensure they are in line with national goals. Principals are responsible for their schools’ results, evaluated on the basis of national goals, but can receive additional assignments from municipal politicians (Skolverket 2013).

Swedish education policy and governance are thus shaped via a complex interplay among actors at several layers. A key element of the education governance system is that all levels should use evaluative knowledge for analysis, comparisons and improvement (Jarl and Rönnberg 2015; SOU 2014:12). Table 1 outlines the prevailing evaluation systems divided into three evaluation objects: students, schools and school providers, and education systems. As the table shows, a substantial number of evaluation systems is in place. Government agencies outside the education sector, such as the National Audit Office, are also involved in evaluating certain aspects of education. Via the National Agency for Education (NAE), Sweden participates in international studies run by organisations such as the OECD, International Association for the Evaluation of Educational Achievement (IEA), and the EU. Moreover, non-governmental organisations with stakes in the education system are increasingly running their own evaluation systems, for example, the Swedish Association of Local Authorities and Regions (SALAR) and the Swedish Association of Independent Schools.

Table 1. Evaluation and evaluation systems in Swedish school governance

Students	Formative assessment (by teachers, e.g. through individual development plans) Student grades (set by teachers) National tests (set by teachers) International tests (also used in evaluation of the education system)
Schools and school providers	Public Aggregated statistics on student performance and other key data (NAE) SIRIS (NAE) SALSA (NAE's tool for local statistical causal analysis) Jämförelsetal (NAE) Evaluation of government policies and programmes (NAE) Schools Inspectorate (SSI) Cohort-sequential longitudinal databases evaluation (Gothenburg University) PESOK (Stockholm University) Systematic quality work (schools, municipalities, independent school providers) Municipal audits and inspections (municipalities) Non-government organisations Open Comparisons (Swedish Association of Local Authorities and Regions) Bästa skolkommun (Swedish Teachers’ Union) Grundskolekvalitet.se (Confederation of Swedish Enterprise, Swedish Association of Independent Schools, Swedish Association of Local Authorities and Regions) A better school (Swedish Institute for Quality) Skolbarometern (Information Tools Scandinavia AB) SIQ Skolindicator.se (HB Educa) Skoldialogen (Catalyt AB)
Education system	Public National statistics (NAE) European Survey of Language Competences (NAE, EU) Key Performance Indicators (NAE, EU) Evaluation of education policies (Institute for Labour Market and Education Policy evaluation) Knowledge brokering (Swedish Centre for Educational Research) Evaluation of government activities (Agency for Public Management) Performance audit of government agencies (National Audit Office) Non-government organisations PISA (NAE, OECD) PIRLS, TIMMS (NAE, IEA) TALIS (NAE, OECD) Education at a Glance (NAE, OECD)

Conceptual and theoretical framework

A policy implies an intended or enacted course or principle of action, adopted by an actor as a solution to perceived problems or to help reach certain objectives and targeting specific groups, areas etc. (Rizvi and Lingard 2010). The term policy space denotes a set of closely interrelated policies in a field, and whose structure includes both the internal arrangements of its policy elements and the linkages and intersections among them (Majone 1989; Wildawsky 1979).

Evaluation systems are structures and processes established to produce streams of data or knowledge (the terms are used interchangeably here) intended to play a role in future action situations. Usually, these systems exhibit certain generic traits. They are ‘owned’ and implemented by actors in organisational structures and institutions from which the evaluation systems’ legitimacy is derived. Evaluation systems also have a certain permanence; they produce input, output and outcome data and are directly or indirectly linked to presumed uses and functions, which in turn suggests the existence of one or more target groups or users (Hanberger 2013; Leeuw and Furubo 2008; Rist and Stame 2006).

In this article, evaluation systems are conceived as integral to a country's education policy space and governance structure within which they are supposed to solve policy problems and fulfil or support governance functions, such as strengthening accountability and facilitating policy and programme improvement. Our focus here is on local school governance, and on how evaluation systems influence municipalities, independent school providers, and schools in managing the accountability and school development functions. Different intended functions are undoubtedly more or less consistent with different types of evaluation systems and knowledge (Hanberger 2013), but evaluation systems are often claimed to be multifunctional and serve various uses and users (Dahler-Larsen 2013).

Establishing evaluation systems for accountability functions essentially entails an institutionalised arrangement between a principal, who is undertaking evaluation to hold an agent to account on the basis of some kind of judgement criteria for expected conduct. The nature of judgement criteria can affect various aspects of conduct, such as finances, fairness and performance (Behn 2001), but the focus here is on accountability for fairness and performance. Accountability for fairness refers to holding organisations, civil servants and professionals accountable for following rules, standards and procedures and for paying due attention to the values to be upheld, such as fairness and equity. Accountability for performance is not about complying with rules, but instead concerns holding governments and organisations accountable for the outcomes and consequences of public policy. However, this is a highly contested issue in a democracy as, according to Behn (2001), it is not only the government that can set the expectations and define the criteria for public policy performance.

The accountability relationship can take various forms in a representative democracy. It can be vertical, for example, when citizens hold governments to account or central government holding local governments to account, and it can be horizontal, for instance when officials and teachers are held to account by parents and the media (Bovens 2010). The accountability relationship can have a ‘soft face’ and merely concern transparency, i.e. data dissemination and access, or a ‘hard face’, including transparency along with the principal's capacity to sanction or reward the agent in various ways (Fox 2007).

Other, quite different conditions, for example, extra resources and evaluation knowledge that is actionable for street-level actors (Grundmann and Stehr 2012), are needed for an evaluation system that is intended to fulfil or support the function of policy and programme (i.e. school) improvement. Improvement refers to aspects of school development such as changes for the better in school culture, teaching, equity and school and student performance. School development understood in this way allows for variation in what are conceived as changes for the better (Hanberger 2016).

Evaluation systems (and policies) in a crowded policy space are likely to interfere with the activities of other evaluation systems (and policies), and with the broader range of education policy and governance in ways that may have unintended consequences. Evaluation systems operating in a crowded policy space may, for example, transform problems intended to be solved. The result of such interference can be described in terms of new policies. Drery (1998) speaks of ‘policies by the way’ arising when policy development and education governance result from ad hoc decisions.

Methods

We initially mapped the population of evaluation systems operating in the Swedish education system with the help of existing reports (Hansson and Hägnemark 2012; OECD 2011; SOU 2013:30) and the findings from other articles in this journal (e.g. Hanberger, Lindgren and Lundström 2016). The results are presented in Table 1. The five systems selected specifically for discussion in this article are indicated in italics in the table.

The selected evaluation systems are: the Information System on Results and Quality (SIRIS), Systematic Quality Work (SQW), Swedish Schools Inspectorate (SSI), Programme for International Student Assessment (PISA), and Open Comparisons (OC). Our first selection criterion was that the selected systems must be authoritative; accordingly, as stated earlier, the five selected systems are ones considered authoritative and dominant in school governance. Three further selection criteria were used, in that each selected system must display variation in system owners, sanctions and target groups. SIRIS, an instrument for local correlation analysis, is also an authoritative evaluation system and one of three benchmarking systems run by NAE. SIRIS includes most indicators used in the other two systems; we selected it because it includes more data on schools and because these data are divided by school owner and school unit. The SSI is a hard evaluation system that relies on sanctions, whereas OC, a voluntary system, is a softer system. SQW is somewhere between SSI and OC, being compulsory but without sanctions. PISA, a global, transnational evaluation system, is by any measure the most authoritative international evaluation system; it covers more competencies than does either TIMSS or PIRLS, which is why PISA was selected.

We describe the systems’ targets groups, the types of knowledge they generate, and the officially stated intended functions of each system. The data employed to do so consist of policy documents, websites, accounts and reports of the evaluation systems, and studies describing the systems. The actual data used are referred to along with the system descriptions. In examining the documents, we make use of qualitative content analysis, asking questions about each evaluation system with respect to the above-mentioned aspects. We then compare these findings and discuss their possible consequences and implications, one by one and together, for local school governance in light of the conceptual and theoretical framework.

Key features and functions of the five evaluation systems

All five evaluation systems examined here are first described individually and then compared in terms of their backgrounds, target groups, types of knowledge produced and intended functions.

Information System on Results and Quality, SIRIS

The NAE is responsible for all statistics reported by all schools and school providers, and the Agency also collects various other types of data. To make the different kinds of data at the NAE's disposal accessible, the Information System on Results and Quality (SIRIS) was set up by the Agency in 2001 as an online database. The key target groups addressed by SIRIS are school principals, municipal politicians and officials, and independent school owners (Skolverket 2015b).

The data are presented at the municipal and school levels and comprise: i) key statistics on schools (e.g. numbers of students and teachers, teacher qualifications, and costs); ii) final-year leaving certificates; iii) student results at national tests; and iv) results of annual surveys of students, staff and parents carried out by Statistics Sweden. To enable fair comparisons between schools, SIRIS is linked to SALSA, an indicator system developed by NAE as a tool for local correlation analysis. SALSA is based on a value-added measure that takes into account school composition in terms of parental education, ethnicity and the gender distribution of students (Hansen and Lander 2009). SIRIS also contains reports on SSI's regular inspections and handling of complaints (e.g. about mistreatment and inappropriate teacher behaviour), documentation of municipalities’ and schools’ Systematic Quality Work (SQW), as well as applications for and follow-ups on government grants.

The intended function of SIRIS is to provide a knowledge base for various analyses, to enable comparisons between schools and benchmarking in order to determine what can be improved. What do we know about student backgrounds? Are our processes of student assessment and grading equivalent and fair? Are we distributing resources in a way that suits student needs? Have the ways we work in our schools been successful? These are just a few issues that, according to NAE, can be analysed using SIRIS data. Another intended function is to give the public a better understanding of school performance by providing balanced information about schools adjusted for their capacity (Skolverket 2015b). Accordingly, SIRIS is predominantly intended to support local school development processes; accountability for performance in terms of displaying the performance of schools to the general public is a secondary purpose.

Systematic Quality Work, SQW

The concept of quality assessment has been a concern of Swedish education governance since the 1980s and was formally adopted in the 1990s (Bergh 2011; Lundström 2015). As of 2010, all schools and their governing bodies are obliged to set up and employ a Systematic Quality Work (SQW) programme for monitoring and evaluating their performance in relation to national education goals. Assessing the attributes of SQW is an important element of the SSI's regular school inspections, an element that is usually severely criticised for its deficiencies (see, e.g. Skolinspektionen 2015a, b).

The NAE has formulated general advice to guide SQW, based on a classic quality management model. In line with this model, SQW should be an on-going cyclic process including planning and realisation, analysis and follow-up with mandatory documentation of all phases. SQW can encompass all kinds of quality work and evaluative knowledge concerning goal achievement, and schools and municipalities can make their own decisions about what kind of data to use. Still, reports reveal that quantitative data, notably from OC rankings, are the most widely used (Blanchenay, Burns and Köster 2014). This implies a focus on subject knowledge requirements while other parts of the curriculum, i.e. overall goals and fundamental values and tasks, are de-emphasised. Local goals can also be included as long as they are in line with the national ones. School providers and principals are regarded as key actors, but SQW is supposed to be an open participatory process involving teachers, students and parents (Skolverket 2012).

The main intended aim of SQW is to promote student achievement and school improvement in line with national educational goals, and it is assumed that SQW is an effective means for doing so (Lundström 2015). SQW, as devised by NAE, is imbued with school development intentions and underpinned by the recurrent use of terms and phrases such as ‘collegial learning’, ‘learning organisations’ and ‘participation of school actors at all levels’. However, the emphasis on goals and documentation is not just a matter of the information needed as a basis for improvement, but is also a way to hold school practitioners at various levels to account. The fact that SQW is a key element of the SSI's regular inspections reinforces this. SQW can therefore be perceived as fulfilling both school development and accountability purposes, although the former is more obvious.

School inspection

School inspection has been a feature of Swedish education since the mid-19th century. The present inspectorate, the SSI, was established as a government agency in 2008, mainly in response to declining school performance and as an adaptation to the growing school market (Gustafsson, Lander and Myrberg 2014). The SSI conducts regular inspections of schools (both municipal and independent) as well as quality audits of smaller samples of schools. The SSI also investigates complaints and approves the formation of new independent schools. As national tests are graded locally by the students’ own teachers, and national assessments have demonstrated that teachers’ grading is often inconsistent, since 2009 the government has commissioned the SSI to re-assess the national tests (Skolinspektionen 2015c). Our focus here is on the SSI's regular inspections and thematic quality audits, which we regard as the most evaluative-like of the SSI's tasks and the most relevant to local school governance.

The basis of the SSI's duties is the agency's interpretation of the Education Act and other national policy documents. Every municipality and school is regularly inspected in terms of several aspects (i.e. goal achievement, leadership and development, learning environment, individual pupils’ rights, and SQW) that together describe what constitutes ‘good education’. These aspects are derived from official policy documents, operationalised by the SSI to make them measureable and observable, and then matched with: data on attainment, grades and test results; reports requested from schools and from municipal and independent school providers; and interview and observation results. A survey distributed to all teachers and students (as well as their parents) in schools subject to regular inspection is another source of data (Skolinspektionen 2015d).

Quality audits are thematic and focus on particular issues, such as educational leadership and the marking of student work. A knowledge framework based on research into the issue at stake is constructed by the SSI, and applied to a sample of schools assessed as to whether they match the framework. The aim of quality audits is to establish a knowledge base supporting improvement of the quality of all schools in line with the knowledge framework for the issue audited (Gustafsson, Lander and Myrberg 2014; Lindgren et al. 2012). The intended function of quality audits is thus school development and the target groups are municipal and independent school providers and their schools.

The practices of the SSI are very much vertically and ‘hard-face’ oriented, their intended function primarily being to enforce accountability for both fairness and performance, though school development is also to be promoted. ‘Juridical’ procedures and techniques are used to assess whether schools are performing well enough, and inspection results are reported mainly in terms of deviation from the criteria used. This is reinforced by the SSI's mandate to shut down or impose economic penalties on poorly performing schools, and by the fact that all reports are published on the Internet and presented to the media (Hult and Segerholm manuscript; Lindgren et al. 2012). The SSI's juridical orientation is particularly evident in its regular inspection activities, while the features of quality audits are oriented to school development. The results of a quality audit explain what needs to be done to improve conditions in order to improve national goal achievement in a specific area.

Programme for International Student Assessment, PISA

The Programme for International Student Assessment (PISA) is a major international student test developed by the OECD to provide policy actors at various levels with trustworthy performance data that can be used for evaluating and benchmarking education systems (Grek 2012; Hanberger 2014; Mangez and Hilgers 2012). PISA is regarded as the key pillar in producing knowledge used to shape educational policy in many countries (Carvalho 2012).

In 2012, PISA provided the following: a profile of knowledge and skills among 15-year-olds in mathematics, with reading, science and problem-solving as minor assessment areas; an assessment of the financial literacy of young people; a background questionnaire seeking information about students as well as their homes, schools, and learning experiences; and a school principal questionnaire covering the school system and the learning environment (OECD 2014). PISA also includes in-depth reports, a video series featuring strong performers and successful reforms in education, policy recommendations etc.

PISA's chief intended function is to develop transnational governance and to support school improvement. OECD/PISA also has an intended performance accountability function, i.e. all school actors should consider their responsibility for their country's performance on PISA. The OECD provides knowledge and recommendations for improving education systems in line with the OECD's mission of striving to minimise social disparities in education and economic growth (Hanberger 2014).

Governments and national agencies are the main target groups of the products of this evaluation system, and all school actors are supposed to consider the country's PISA results and the OECD's suggestions for improving education systems. Although PISA does not provide results on how schools perform at the local or individual school levels, municipal education committees, principals and teachers are supposed to take the national PISA scores and the OECD's policy recommendations into account when developing their schools (Hanberger 2014).

Open Comparisons

The Swedish Association of Local Authorities and Regions (SALAR) is legally a non-profit organisation representing the interests of Sweden's municipalities and county councils. Since 2007, SALAR has provided Open Comparisons (OC), i.e. comparable municipal-level data on public services in many areas, including education, targeting all actors involved in local school governance.

Each municipality is ranked according to a number of indicators that increased from 8 in 2007 to 35 in 2014. Most SALAR data come from existing databases, particularly those of SIRIS and Statistics Sweden, and include measures of student grades, national test results, teacher competencies, key economic figures, and basic statistics about municipalities (e.g. population and average income). Other indicators concern student views of schools and education based on a specific set of survey questions integrated into the municipalities’ own user satisfaction surveys (the survey is not carried out by SALAR itself) (SALAR 2010, 2015). When reporting the data, municipalities are ranked in a way that resembles the educational league tables found in many other countries.

OC aims to increase the transparency of education results and costs, stimulate benchmarking and induce municipal and independent school providers to improve the performance of their schools. In the most recent report, SALAR (2015) also provides a handbook, based on OC data, to assist municipal school providers in their task of implementing SQW.

The most salient intended function of OC is undoubtedly school development. This is how the managing director of SALAR describes the use and targeted users of the data provided: “Politicians, public officials, principals and teachers may use the results to deepen the analysis of school results and discuss measures for improvement” (SALAR 2015, 7). A function promoting accountability, vertical as well as horizontal, is less salient, but is at the same time clearly evident in the way SALAR deliberately engages in publicly ranking municipalities according to how well their schools perform in relation to SALAR's interpretation of national goals. This may well be seen as a way of passing judgement, although schools and municipalities are not accountable to SALAR in any formal sense.

Many systems doing almost the same thing

Evaluation systems are evidently crucial elements of the Swedish education system and can be seen as the glue that, as part of a goal- and result-oriented management system, is intended to join together the various levels of the education system. Our general inventory of evaluation systems indicates that about 30 systems are in Sweden (Table 1), half of which are run by public agencies and half by national and international organisations. Five systems were further examined, paying attention to the type of evaluative data produced, target groups, and intended functions.

Table 2 shows that the five studied evaluation systems collect and report a range of evaluative data of various types, but standardised quantitative data are unquestionably the most common type provided by all five systems. SIRIS incorporates some elements of qualitative data, as does PISA and SSI (which in the table is divided into regular supervision, RS, and quality audit, QA). Schools, as well as municipal and independent school providers can make their own decisions about what kinds of data to use for SQW (which the parentheses in the table are intended to indicate). Reports reveal, however, that while SQW may occasionally include qualitative programme evaluations, schools and school providers tend to rely heavily in their quality work on standardised quantitative data supplied by OC (Blanchenay et al. 2014; Skolinspektionen 2015b).

Table 2. Overview of data emanating from the five evaluation systems

			SSI
	SIRIS	SQW	RS	QA	PISA	OC
Figures on student grades	✓	✓	✓	✓		✓
Figures on eligibility for upper secondary education		(✓)	✓	✓
National test results in specific subjects and grades	✓	(✓)	✓	✓
Key statistics about school costs, staffing, etc.	✓	(✓)	✓	✓		✓
Results of survey on staff/student/parents’ opinions (conducted by SSI/Statistics Sweden)	✓	(✓)	✓	✓
Documentation of local systematic work	✓	✓	✓	✓
Results of survey on staff/student/parents’ opinions (conducted by schools/municipalities)		✓				✓
Results from local programme evaluation efforts		✓
Test results of students’ knowledge in particular areas	✓	(✓)			✓
Tested students’ background survey					✓
Results from school principal survey					✓
Policy recommendations					✓
Reports on schools’ and school providers’ shortcomings in inspected areas	✓		✓
Findings from stand-alone evaluations on particular issues from a sample of schools	✓			✓
Reviews of existing scientific knowledge for particular issues	✓			✓
Applications and follow-ups of government grants	✓

Not only do the five evaluation systems mainly contain quantitative data, but three types of quantitative data predominate: student grades, student test results, and student/parent satisfaction surveys. Student grades and national test scores are compiled and presented in different ways but for nearly the same purpose by three or more evaluation systems. Student/parent satisfaction surveys with fairly similar outlines and contents, conducted by Statistics Sweden and SALAR, respectively, are used by four evaluation systems. The quantitative data contained in several of the evaluation systems are, moreover, collected largely from the same data source, i.e. SIRIS, which stores most data at the disposal of NAE.

Table 3 shows that the data provided by the five evaluation systems are directed towards several target groups: national government and agencies, schools (i.e. principals and teachers), municipal and independent school providers, and students and their parents. Schools, municipalities and independent school providers are targeted as users by all five evaluation systems.

Table 3. Overview of target groups and intended functions of the five evaluation systems

			SSI
Target groups/evaluation systems	SIRIS	SQW	RS	QA	PISA	OC
National government and agencies	✓		✓	✓	✓
Municipalities and independent school providers	✓	✓	✓	✓	✓	✓
Principals, teachers	✓	✓	✓	✓	✓	✓
Students and their parents		✓			✓
Intended functions
Accountability for fairness			✓
Accountability for performance	✓	✓		✓	✓	✓
School development (school and student performance)	✓	✓	✓	✓	✓	✓

The table also shows that the key intended functions of each of the five systems, as we have interpreted the consulted policy documents, is school development, while two systems are multifunctional. SIRIS, SQW, the SSI's quality audit, PISA and OC all primarily aim to support improvements in the performance of students, schools and – in the case of PISA – the national education system. The SSI's regular inspections with their clear focus on accountability for fairness are an exception to this pattern. PISA has an intended accountability for performance function, as well as a development function, in that all school actors are supposed to consider their responsibility for their country's PISA performance.

How the data provided by the evaluation systems should or can be used for school development is very much left to the target groups to decide, even though SIRIS, SQW, PISA and OC come with succinct guidelines, and the SSI's regular inspections with to-do lists. School development can imply improvements in school culture, teaching, equity, and school and student performance. However, the guidelines and to-do lists indicate that target groups are encouraged to take actions that chiefly address school development as constituting student and school performance.

As indicated, the SSI's regular inspection aims at accountability for fairness while PISA aims at accountability for performance. Accountability, for fairness as well as for performance, can have a ‘soft face’ or a ‘hard face’ depending on whether or not it involves sanctions. Accountability with a soft face (Fox 2007) can be understood as an implied function. This means that one or more accountability relationships can be identified between a principal and an agent, for example, between NAE and schools, municipalities and independent school providers. These agents are mandated to disclose and deliver certain information about their conduct. Yet no formal sanctions exist for agents whose conduct is not up to standard, which is what distinguishes a hard from a soft face of accountability. However, as the evaluative data produced are publicly available, accountability for performance and informal sanctions may be regarded as enforced through the “power of shame” (Fox 2007, 665) and perceived as relatively hard.

The only evaluation system with a distinct hard face accountability function is that of the SSI's regular inspections. These inspections are conducted within a framework of general standards, the contents and requirements of which the schools, municipalities and independent school providers are well aware of in advance. The SSI reports indicate whether specified standards have been met and shortcomings to be dealt with, i.e. accountability for fairness. Inspectors follow these reports up, and the SSI has the power to fine schools that have not undertaken remedial action.

Consequences of evaluation in a crowded policy space: concluding discussion

We now return to the opening argument of this article, namely that evaluation systems in a crowded policy space like the Swedish one may interfere with each other in ways that can have unintended consequences, even transforming problems intended to be solved and creating new ‘policies by the way’ that are not the result of intentional policy decisions. How can our examination of the five evaluation systems described in the previous section be understood in relation to this argument?

Inherent in the notion of evaluation systems as policies, applied here, is the assumption that those systems are set up by actors to help solve certain social problems. An evaluation system is probably seldom depicted by its inventor or owner as a ‘solution’ to a specified social problem, but built into any evaluation system are more social problems that the system is supposed to help resolve. Evaluation systems can thus instead be conceived as ‘solutions’ to governance problems and as indispensable components of school governance.

The risk that SIRIS, SQW, SSI, PISA and OC may interfere with each other and transform the problems that the individual evaluation systems are intended to solve is, at first glance, not very likely as nearly all pursue the same intended function. School development is, except for the SSI's regular inspection activities, the main intended function of all examined evaluation systems. The systems are, moreover, presenting the same core of quantitative data (student grades, student test results, and student/parent satisfaction survey results), which reduces the risk of conflicting expectations from different evaluation systems. However, the multiplicity of evaluation systems could, in our view, still give rise to unintended consequences, some of which can be described as promoting ‘policies by the way’, multiple accountability problems, and increased administrative burdens.

Scaling down the school law and curriculum

In each case, the meaning of school development is only vaguely defined, remaining open to interpretation by the target groups. Yet the evaluation systems’ common focus on a similar and limited quantitative data set, plus the encouragement in the guidelines and to-do lists accompanying that most evaluation systems to comply with predetermined performance goals and data, helps shape the target groups’ notions of what school development is or ought to be. School development therefore becomes closely associated with improvements in student grades, student test results and student/parent satisfaction, while school development in a wider sense concerning aspects such as equity and democracy is ignored. That several evaluation systems point in the same direction adds to the scaling down of the school law and curriculum and, in doing so, leaves little or no discretion to local school governance. With its emphasis on an even narrower range of aspects of education that to a very limited extent capture the goals of the Swedish education system, and by putting pressure on national and local actors to promote reforms and programmes in line with an OECD-oriented education discourse (Hanberger 2014), PISA adds to the described ‘policy by the way’.

While data quality is not a key issue addressed here, and the limited data on which this article is based certainly do not allow for a general conclusion, we nevertheless believe it is important to question the extensive use of student grades and national test results because they so obviously constitute pillars of the Swedish education evaluation system. Student grades measure knowledge requirements for subjects and courses, and national tests were originally designed to help teachers calibrate grades, but both grades and tests tend to serve several other evaluative functions. In our view, it is questionable whether grades and tests are valid measures of performance at the school, municipal and national levels. The assumption that OECD/PISA is a valid test for measuring the quality of education systems can also be questioned (Hanberger 2014). Well-founded evidential criticism of the relevance of using student grades and national tests for various purposes can, furthermore, be found in other studies. Due to inconsistencies in teachers’ grading of students in school relative to their national test results, student grades are not fit for the purpose of evaluation at the school, municipal and national levels (e.g. Gustafsson, Cliffordson and Erickson 2014).

Data quality is not only about construct validity, but is also affected by how data are collected, entered, stored and managed in evaluation systems. The fact that schools as well as municipal and independent school providers bear the primary responsibility for collecting and entering most of the data raises concerns about the potential incentives for local employees to game evaluation systems by manipulating data to make their performance look good (Lindgren 2014). That teachers set grades and score their own students’ achievements in national tests also raises, as indicated above, concerns about the reliability and equity of this kind of data (Gustafsson, Cliffordson and Erickson 2014). Moreover, the question of attribution is not considered in any of the evaluation systems that encompass data on inputs, activities, outputs and outcomes. The cause-effect relationship is, however, implied in the measurement data provided. For example, it is taken for granted that outcomes such as the level of student grades and national tests are due to activities and outputs in the school, municipality or education system, which may not always be the case.

Multiple accountability problems

The five examined evaluation systems (as well as the remaining 29 operating in the Swedish education policy space) are owned and run by a number of actors (i.e. central government agencies, municipal and private school providers, and national and international organisations) that have undoubtedly planned the systems according to their own goals and ends. This multiplicity of owners can itself create multiple accountability problems (Schillemans and Bovens 2011), i.e. uncertainty among target groups as to which evaluation system is supposed to do what, for whom, and on whose authority. Various actors in the target groups can interpret the features and functions of an evaluation system, and come up with competing arguments as to what the data mean and how they should be acted upon. The complex and multilevel Swedish education governance system adds to the opacity of the linkages, as does the increasing involvement of private evaluation actors.

The owner of OC, SALAR, is legally a non-governmental organisation, not a government agency. Target groups such as municipalities, independent school providers, and schools are therefore not accountable to SALAR in any formal sense. SALAR has no mandate to hold anyone to account on the basis of OC or in any other way, for that matter. Yet SALAR has defined criteria for good conduct and ranks municipalities according to how well their schools perform in relation to those criteria. In spite of this, OC is read by politicians and officials at all levels and is described as the predominant source of information in the target groups’ own evaluations (i.e. SQW) of goal attainment (Blanchenay et al. 2014; Skolverket 2013). Other examples outside the five evaluation systems examined here are those of the Swedish Association of Independent Schools and the Confederation of Swedish Enterprise (Svenskt Näringsliv 2013). These bodies have engaged in evaluation by establishing their own quality standards, using available official statistics together with their own data to participate in the school quality discourse and to hold schools accountable for their performance.

Evaluation of education implies the systematic assessment of the quality or value of something, in the present case, students, schools and school providers, and education systems. Evaluations and evaluation systems are hence by nature encouraging the search for flaws and failures. In a situation like the Swedish one, with a large number of evaluation systems and actors, there is a great likelihood that all these fault-finding missions and accountability holders will generate a climate of negativity (Schillemans and Bovens 2005) that diverts attention from the evaluation systems’ main intended function to support school development.

Increased administrative burden and the creation of new intermediary job functions

Multiple evaluation systems require a lot of data, as well as people and arrangements to handle the demands originating from the evaluation systems. The design, implementation and maintenance of an evaluation system require substantial resources on the part of the system owner, and target groups must collect and deliver the data needed. Once the data have been provided by the evaluation system, to have meaning they need to be read, analysed and used by schools, municipal and independent school providers, and other target groups. These tasks require resources and substantial expertise and, according to reports, have led to an increased administrative burden and the creation of new intermediary job functions (Hall 2013; Proposition 2012/13:195; SOU 2014:5). How such arrangements find expression in Swedish local education governance is still very much an open question (Sandén 2015). Still, the number of municipal administrators with titles such as controller, quality assurance coordinator, and managers, has grown significantly over the last two decades (Montin 2015).

Finally

At first sight, the Swedish education policy space appears to be characterised by multifunctional evaluation systems serving various uses and users. However, our examination of the five dominant systems in this article indicates that the policy space could better be described as one crowded by multiple public and private evaluation systems, many of which are doing almost the same thing. All five systems collect and document streams of knowledge of a similar quantitative type, there is a considerable data overlap in that three of the five systems (SIRIS, SQW and OC) address the same local school actors with almost the same data to hold them accountable for performance with the primary goal of supporting school development conceived as improved student and school performance. Because of this apparently crowded policy space, with overlapping and uncoordinated evaluation systems, it is not easy to see how, if at all, the systems can work together without creating multiple accountability problems and confusion about to whom teachers and principals are accountable. Further, holding teachers and principals to account for how students perform rests on the assumption there is a causal relationship between what teachers and schools do and how students perform. However, other factors such as students’ family backgrounds matter more (Levin 2010).

We need a discussion about what local school actors can and should be accountable for, and about which evaluations truly support school development, including aspects of school development such as improving teaching and the school culture.

Additional information

Notes on contributors

Lena Lindgren

Lena Lindgren is Associate Professor at the School of Public Administration, Gothenburg University. She works with teaching and research focusing on evaluation and policy analysis, and has also conducted several evaluations on commission for government agencies, particularly in the field of education. Email: [email protected]

Anders Hanberger

Anders Hanberger is Professor in Evaluation and director of research at the Umea Centre for Evaluation Research, Umea University. His research areas cover policy analysis and evaluation research and focus on public policy, governance, evaluation, evaluation systems, and methodology. Email: [email protected]

Ulf Lundström

Ulf Lundström is Associate Professor at the Department of Applied Educational Science, Umeå University. His main research interests lie within the areas of education policy, e.g. focusing on marketisation and inclusion, evaluation and the teaching profession. Email: [email protected]