Data Analysis for School Improvement within Coupled Local School Systems: Which Data and with What Purposes?

ABSTRACT

From a new institutional theoretical perspective, this article explores school actors’ sense-making linked to data-based decision making (DBDM) policy in general and processes of data analysis in particular. The study revealed how actors’ interpretation of and response to DBDM pointed to strong and weak couplings between and within the local school system’s organizational levels. While teachers emphasized informal, daily analyses, the LEA and principals placed importance on formal, district and school-based analyses. In the same way teachers to a greater extent think that too much resources is spent on collecting and analyzing data rather than on innovation and school improvement.

Introduction

In many school systems around the world, there is an increasing focus on the quality of schools and their pupils’ academic achievements. This has resulted in an increase in control over pupils’ performance and in demands for school actors and decision-makers to improve the institutions for which they are responsible. Consequently, school improvement and effectiveness have become a part of high politics and an important issues at all levels of school systems worldwide. Several studies (e.g., Lai et al., 2014; Poortman & Schildkamp, 2016; Van Geel et al., 2016) showing how effective data use by school actors can reinforce school improvement have precipitated an international policy movement toward data-based decision making (DBDM). That is, DBDM is seen as a powerful way of improving schools in many countries.

DBDM can take place at several levels within the school system, from the classroom level up to the policy-making level. This means that data can be used for several purposes: decision-making linked to school policy, resource allocation, school development, and classroom pursuits (Schildkamp & Kuiper, 2010). Though the context in which it is applied may vary, the overriding aim of DBDM is the same: to make schools’ results more transparent and decision-making linked to school improvement more effective. The central assumption within DBDM policy is that access to data, in combination with accountability, will reinforce school actors’ professional development and improve educational practices. Nevertheless, the line separating use of DBDM as a means of school development or external control is thin and therefore important to pay attention to, especially when it comes to questions of legitimacy and school actors’ motivation for using data (Sun et al., 2016).

A crucial part of school actors’ data use is the process of analyzing different forms of data. Even if DBDM occurs at different levels and in different subsystems within the same local school system, these processes are closely linked and highly dependent on one another (Adolfsson & Alvunger, 2017). Although DBDM has been extensively researched, we know little about how school actors at different organizational levels within the local school system understand the data analysis process linked to DBDM, nor how they value different forms of data (Prøitz et al., 2017; Schildkamp, 2019). Based on the results of a three-year research project in a large municipality of Sweden, the overall aim of this article is to explore educational actors’ (teachers, principals, and officials of the local education authority (LEA)) sense-making linked to DBDM, with a specific focus on the processes of data analysis and the data used therein. Accordingly, this study may provide crucial insights into understanding central factors linked to school actors’, at different organizational school levels, enactment of DBDM policy.

School Actors’ Data Analysis and Different Forms of Data

Several processes and actions constitute school actors’ use of data (Sun et al., 2016). However, a central process and the focus of this study is the analysis of different forms of data, i.e., the process in which school actors determine which data should be taken into account, make meaning of that data, and come to decisions or take actions as a result. Schildkamp and Kuiper (2010) describe this process of data analysis in terms of: “systematically analyzing existing data sources within the school, applying outcomes of analyses to innovate teaching, curricula, and school performance, and implementing (e.g., genuine improvement actions) and evaluating these innovations” (p. 482).

This analysis process of noticing, interpreting, and extracting implications for action is framed by a number of different factors such as individual beliefs, knowledge, and motivation; but also by external conditions like school organization, leadership, and culture (Hoogland et al., 2016). It follows that school actors will likely filter the data through their own lenses and experiences, which implies that the same data can yield different meanings for different people, especially between different groups or levels within a school system (Datnow et al., 2017). As empirically shown by Kahneman and Frederick (2005), this process of filtering can cause school actors to adjust the outcome of an analysis to confirm their preexisting assumptions and beliefs, meaning that alternative interpretations and explanations must subsequently be sorted out.

School actors often have access to multiple types of data. Accordingly, the selection of data to be analyzed is central to the process of analysis (Mausethagen et al., 2018). It is therefore important to note that different school actors can account for or value data in different ways depending on the type of analysis to be conducted, the purpose behind the analysis, and who they are (i.e., a teacher, principal, or policy maker). Ikemoto and Marsh (2007) make an analytical distinction between different forms of data in terms of input, process, context, and output data. Input data comprise data that is linked to student demographics, while process data contain information about education processes, i.e. instructional practices. Context data, in turn, comprise information about policies, external conditions and different forms of available resources that can affect the schools’ outcomes in different ways. Finally, output data, which is described as the most common form of data, includes information about students’ achievements and goal attainment (Ikemoto & Marsh, 2007). Data may also be distinguished as being either formal or informal. In this article, formal data is defined as data that is standardized and often produced externally of the teacher, regardless of whether she or he wants it or not, for instance, national test data, students’ grades, School inspectorate assessment data etc. In contrast, informal data is understood as data that is created by the teacher in close relation to the students and the teaching practice (e.g., results from own produced tests, lessons planning, conversations with students or parents, documentation from classroom-observations etc). Accordingly, compared to formal data, this kind of data can be characterized as internal and non-standardized (Datnow et al., 2020). Previous research has pointed out that teachers seem to prefer informal data in the data analyzing processes, because it allows for an immediate “pulse check” of their teaching. In contrast, standardized and large-scale data were considered much more difficult to analyze and were therefore more often used as background information (Mausethagen et al., 2017; Sun et al., 2016). In the following section, the study’s theoretical points of departure will be outlined. This will enable a more specific formulation of the research questions that guided this study.

School Actors’ Sense Making within Loosely-Coupled Systems

In this paper, we apply a theoretical framework inspired by organizational theory in combination with theories of policy implementation (Coburn, 2001; Orton & Weick, 1990; Spillane, 2012). From this perspective, how DBDM policy will play out at the local school level may be anticipated based on a combination of two factors: the relationship between different levels and subsystems within an organization; and school actors’ experiences and perception of DBDM policy. National school agencies, LEAs, school leaders, and teachers constitute the main organizational levels of what is usually referred to as the school system. From our perspective, the relationships that exist between these organizational levels, or subsystems, is of great interest (Spillane et al. 2011; Adolfsson & Alvunger, 2017, 2020). The concepts of loose coupling, tight coupling, and free coupling have been used here as a lens through which these relationships may be viewed to better understand processes and outcomes within the school system (Orton & Weick, 1990). The concept of coupling is useful in the way it put focus on dimensions of distance and proximity within an organization’s different subsystems, as well as to qualities of being open or closed in relation to other organizations.

An organization, e.g., a school, in which couplings between its different subsystems are tight, can be described as being responsive without distinctiveness. This means that the school responds to external influences while the boundaries between its different subsystems are characterized as weak and indistinct. In a loosely coupled system, the school would be characterized by both responsiveness and distinctiveness, which implies that it is responsive to external influences but with clear and distinct boundaries between its internal subsystems. Finally, in a de-coupled system, the school would be characterized as having distinctiveness without responsiveness. That is, clear and distinct boundaries exist between its internal subsystems, and the same subsystems are not responsive to external pressure (Orton & Weick, 1990, p. 205). A study based on these concepts implies that focus is affected by when, and in what ways, couplings within an organization are strong or weak. This, in turn, is of importance when it comes to understanding policy processes in terms of how actors in different subsystems (for example, at different organizational levels in the local school system) understand and act in relation to (or make sense of) a specific policy.

Several studies have pointed out that the school system in general is characterized as a loosely coupled system. This implies that within a school system, multifaceted goals, means, and variances in the definition of problems and priorities are often found. Consequently, such organizations become difficult to coordinate and control, not least because the knowledge bases, experiences, priorities, and needs of actors within different school subsystems are not necessarily in harmony with central policy directives and intentions. This may in turn explain why, for example, educational reforms are often not implemented in line with policy makers’ original intentions (Spillane et al., 2011). Previous studies (e.g., Adolfsson & Alvunger, 2017) have similarly shown that school development initiatives are unlikely to be successful unless they actively engage and recouple the involved subsystems within the local school system.

Consequently, how school actors within different subsystems will respond and act in relation to a specific policy, for example, a DBDM policy, will depend on their repertoire of knowledge, experiences, and understanding of the issue (Spillane et al., 2002). Expressed in another way, insights into school actors’ sense making become crucial to understand the outcome of a DBDM policy: “people generate what they interpret” (Weick, 1995, p. 34).

However, from a theoretical point of departure in terms of school actors’ sense making within a loosely coupled school system, DBDM implementation is not just dependent on school actors’ knowledge, understanding, and capacity, but also on will and organizational factors such as leadership structures, groups, and routines etc. (Coburn, 2001; Spillane et al., 2002). From that perspective, a central question appears concerning the relationship between DBDM policy and school actors’ use of data in practice. Specifically, based on the results of a three-year research project, we explored school actors’ at different organizational levels within a local school system in terms of LEA officials, principals and teachers, sense making linked to DBDM in general and of the data analysis processes in particular. The following research questions guided our study:

RQ1: How do school actors at different organizational levels within the local school system understand the aim of DBDM?

RQ2: What are the same school actors’ experiences and perceptions of the analysis processes involved in DBDM?

RQ3: What kind of data are used at the different levels, and which are most valued in data use settings?

Methodology

Research questions RQ1 to RQ3 were answered through a case study (Bryman, 2003; Yin, 2018). The data used in this study were collected within the scope of an ongoing evaluation research project. Over three years, we thoroughly studied and analyzed the LEA in a major municipality in Sweden during its implementation of a revised quality system at six comprehensive schools. These six schools were selected based on that they, all together, were representative of schools in the current municipality. That is, the selected schools were situated in different parts of the municipality, in the center of the town as well as in the suburbs, also representing a range of socioeconomic levels among the schools. In addition, four of the schools enroll students from preschool class to grade 9 and two have pupils from preschool class up to grade 6. Finally, the six schools represented different school sizes, with a variation of 285 and 800 students. The research design, which was inspired by a multi-method approach (Creswell, 2010), was explorative in nature and allowed us to obtain different but complementary types of data (Creswell & Plano Clark, 2007). A sample of this data has been used in the current study. In the following section, we will describe this data more specifically, including the method considerations that lay behind data collection.

In the first step of the study, an analysis of local policy documents linked to the LEA’s quality assurance system (e.g., core documents on the organization of local schools, on framework describing the LEA’s quality management policy, and on leadership and management structures) was conducted. The aim of this document analysis was to gain a deeper contextual understanding of the LEA’s formal quality system, governing organization and infrastructure for communicating results and controls in relation to schools. The analysis revealed important insights and knowledge that in turn created a base for subsequent interviews with school actors at different organizational levels. Results from the document analysis are briefly presented in the first part of the results section.

Subsequently, semi-structured interviews were conducted with LEA-officials (13 individual interviews), principals at six schools (6 individual and 4 focus group interviews, with a total of 16 principals), and teachers at the same six schools (12 focus group interviews with a total of 47 teachers). All school leaders at the six schools were included in the study. Regarding the teachers, we were interested in those with experience of, and actively involved in, the school’s improvement work, resulting in a selection co-created in dialogue with the school leaders. All the interviews were carried out using an interview guide, thematically organized into three main areas: a) the relationship between the LEA’s quality system and that of the schools; b) the LEA’s or the schools’ organization and processes linked to data use, data analysis, and identification of areas for improvement; and c) school actors’ experiences and perceptions of DBDM. Each interview of approximately 60 minutes was recorded, transcribed, and analyzed.

The process of analyzing the empirical material was conducted in two steps. In the first step, the transcribed material was analyzed exploratively. Based on the research questions, the aim of this step was to identify and illuminate patterns as well as variations between the organizational subsystems, but also within the subsystems themselves. In the next step, these patterns and variations were analyzed in light of the study’s theoretical framework, i.e., in terms of tightly and loosely coupled systems and the sense-making concept. That is, the aim was to illuminate how the local policy linked to DBDM played out in different subsystems as school actors co-constructed the policy in their different settings.

A central part of the research project consisted of recurrent occasions of discussion regarding the result of the study with the school actors at the LEA and the six schools. The purpose of this was twofold; firstly, it enabled the school actors to get important insights and useful information about their own quality system and the improvement processes they were involved in, secondly, these discussions worked as, what Kvale and Brinkman (2014)term as, a sort of respondent validation. That is, through these discussions we got a continual response directly from the school practice if the results from our study were regarded as valid or not.

A Description of the Case

In this section, we briefly contextualize the case. In a first step, we give an overall description of Swedish educational policy. This description constitutes the foundation upon which, in a second step, we may begin to discuss the LEA in question and its quality assurance system.

The Swedish school system has been characterized by far-reaching decentralization since the early 1990s. This means that municipalities and independent school heads have considerable authority over the schools for which they are responsible. However, Swedish students’ poor academic achievement over the past two decades has spurred a recentralization process. This has included, for example, a reformed Education Act issued in 2010 that: emphasized local authorities’ responsibility for equity and student achievement; strengthened principals’ authority; introduced a new standard-based curriculum for compulsory and upper secondary schools (Cosner, 2011); implemented a national school inspectorate to audit and monitor schools; and initiated professional development programs on a national level. In addition, in light of such a policy shift, the emphasis on school quality assurance and accountability has grown stronger at both the intermediate level (LEA/district level) and individual school level.

According to the Swedish Education Act, 2010, a school must be managed and coordinated by its principal. The principal acts as an educational leader and is responsible for working in accordance with and evaluating results in line with national goals. In abiding by the ordinance of the act, all school organizers, schools, and preschools must implement quality assurance measures in a methodical way by “systematically and continuously planning, following up, and developing education” (, 2010 :800, ch.4, s.3). The LEA, together with individual schools, are held accountable for ensuring that education is aligned with the national goals as well as legal requirements and school ordinances. The superintendent at the LEA level has the operational responsibility of leading principals, equitably distributing resources, and achieving national education goals.

Alongside the Education Act, the Swedish National Agency for Education gives specific directives and recommendations to the schools regarding how Schools and LEA’s should organize, structure and carry out their quality assurance. This implies a high degree of uniformity, at least on a general level, between the Swedish Schools and LEA regarding structures and processes linked to the quality systems. Consequently, the quality system in focus for this study is, from that perspective, not unique, which means that the results can be regarded as relevant and valid in a broader context.

The case used for this study is a LEA of one of the largest municipalities in Sweden. The schools in this municipality are characterized by high numbers of students with differing ethnic backgrounds. In addition, the municipality has received some of the highest numbers of immigrants to Sweden in the last 10 years. Differences between low- and high-performing schools in the region are also significant, which in turn has put the question of equity high on the local political school agenda.

Findings

Findings are presented following the three research questions. Accordingly, the aim of this section is to explore how school actors’ made sense of a new policy at different organizational levels (subsystems) within a local school system regarding: 1) the aim of DBDM; 2) the process of analysis related to DBDM; and 3) the use and value of different kinds of data linked to DBDM.

Findings from the document analysis and interviews show that against a backdrop of declining academic performance in the municipality, population segregation, and increasing differences in results between schools, the LEA has developed and implemented a revised system for quality assurance and school development. This quality system consists primarily of two components: a professional development program for all school leaders (including the LEA); and the establishment and implementation of new structures, activities, and routines for monitoring results and quality assurance. The professional development program has been conducted within the framework of International School Leadership (ISL) based in Ontario, Canada, with training modules dedicated to school system leaders and school leaders alike. In terms of the LEA’s results monitoring and quality assurance programs, four different mechanisms have been put in place: 1) setting of priorities and decision making; 2) leadership; 3) measurement of learning and results; and 4) organization and strategies for school improvement.

In addition to the introduction of mechanisms for reaching a set of objectives, the LEA carried out a number of activities. In the interviews as well as in the policy documents, the most important of these activities was the dialogue between the LEA and school principals. The starting point for these dialogs, but also other similar DBDM activities at all organizational levels, was the schools’ results and development areas based on the objectives in the national curriculum. As in all LEA activities, different forms of data constituted a fundamental part of this dialogue. This data provided the school actors with an, as far as possible, objective, and informative picture of a phenomenon or issue linked to the school practice. The processes for which school actors determine which data should be taken into account, and how school actors come to understandings about how data is indicative of actions that must be taken, constitute central components of the LEA’s current quality assurance system.

Multifaceted Perspectives of the Aim of DBDM between and within the Local School System’s Different Organizational Levels

When LEA officials described why schools and the LEA itself were using data in their decision-making processes, two main perspectives emerged. The first perspective can be described in terms of control and quality assurance. That is, statistical data enabled the LEA to not only perform a trend analysis of schools’ results, make comparisons between schools, and evaluate and make decisions about school improvement initiatives, but also to set principles’ wage rates. As LEA Official 1 stated: “we control the schools through evaluation, however the most important is that we don’t tell them exactly what they should do, we are primarily interested to know on what basis, which data, they have made their decisions.”

Another representative from the LEA emphasized that to get better control over schools’ results, a well-functioning quality assurance system, at LEA and school levels alike, were required. Furthermore, the same officials stated that an important ambition of the LEA quality system was to increase the uniformity of all schools’ internal quality systems, both in terms of processes and organization. Such uniformity would provide the LEA with access to more standardized school data, which in turn would allow it to make more accurate and meaningful comparisons between school results. As LEA Official 4 stated, “the ambition is to obtain comparable data from the schools that we can use at the LEA level.”

The second perspective that emerged can be described in terms of school development and formative assessment. For example, officials emphasized that different forms of data, both quantitative and qualitative, comprise important knowledge sources for teachers’ professional learning and for building school capacities, saying: “we (the LEA) will help the schools to build up a capacity in their data use, in that way we want them to learn how to identify what they need to develop in their organization” (LEA Official 3).

In addition, representatives from the LEA pointed out that using data many times fortified communications between the LEA and schools. According to LEA officials, the data enabled a much more focused and relevant discussion about the schools’ core business. Correspondingly, by providing schools with different forms of data, the LEA also helped to strengthen principals’ capacity to lead: “we have much more data now and many of the principals are inspired to use it, they have acquired a lot of input from us linked to how they can use data in their leadership” (LEA Official 2).

Another LEA Official emphasized, in the same way, that data and principals’ data use enabled situations for professional learning between the principals:

I have organized my principals in so-called principal-groups. In these groups, the principals can support each other and share experiences of school development and school leadership. That is, it is an important context for professional learning; in the last meeting, for example, we had a discussion about how to follow up and analyze students’ results (LEA Officials 1).

Accordingly, a number of perspectives regarding the aim of DBDM could be distinguished in the interviews with the LEA officials. Regardless of perspective, all LEA officials emphasized the importance of using data as a mean for making evidence-based decisions. Similarly, we found multifaceted understandings of DBDM aims in interviews with principals. However, compared to LEA officials, principals placed a greater emphasis on the fact that data could be used for internal school development. For example, several principals described how different forms of data had become an important resource when it came to analyzing current situations, mapping the school’s results, clarifying pre-conditions in terms of economic and human capital, and finally identifying the school’s development needs, saying: “it is for our own sake that we assess and measure. By using data, we have been able to conduct much deeper analyses” (Principal 3).

Principals described how data was, in some ways, synonymous with valid knowledge. Using data in their decision-making became an important way of providing evidence of the efficacy of school improvement processes, especially when it came to deciding upon initiatives to be undertaken: “from my point of view, we previously based our decisions on loose assumptions. However, now we want to ensure that we do not make such assumptions” (Principal 5).

Although principals primarily described using data for development purposes, we also found examples of them linking data use to external control and assessment. Some principals stated that in their communications with the LEA, it was important that they ensured and demonstrated how they had control over their school, and that based on relevant data, they could provide a rationale for their decisions. As Principal 4 put it, “you cannot come to a quality dialogue with an LEA manager, and not have control over the data linked to your school.”

In sum, among principals and LEA officials, the dominating perception of DBDM was positive. However, among teachers, a more critical view of DBDM appeared. Many of the interviewed teachers discussed data use in terms of control and monitoring, primarily serving external needs. The aim of collecting and analyzing data was therefore perceived to make the school more measurable:

It feels like our municipality has decided that everything must be measurable. And we must act according to this. However, how can you measure soft values in a good way? It is confusing … is it right or wrong? I don’t know how to handle this? (Teacher 2)

However, even if we elicited a number of critical views of DBDM among teachers, several of them seemed to accept the central position that data occupies in the current school improvement system. Some teachers indicated that DBDM could be used toward local school development and improvement. For example, they commented that data was necessary to getting a valid picture of schools’ results and development needs, and could accordingly be an important tool when it came to setting priorities and making decisions about school improvement programs.

At the same time, other teachers stated that data use was something that did not concern them very much. That is, collecting and analyzing data was primarily an issue for the principals to handle. This can be understood in light of an uncertainty that emerged among some of the teachers concerning the degree to which data use could actually support them in their work. One teacher explained this in terms of a lack of skills linked to school improvement and DBDM, saying,

I don’t think everyone has the necessary ‘tools’ when it comes to using data … they just act. It is a shame, because you may lose the potential of data use; in contrast, you may be frustrated. Why are we doing this? (Teacher, 6)

In sum, among teachers, we identified tension in the form of perceptions of data being used to control (the dominating perspective) and data being used to develop. This differs from understandings expressed by LEA officials and principals. In general, teachers were not resistant to data use, but they emphasized their lack of necessary knowledge and skills linked to data use, especially when it came to the processes of analysis.

School Actors’ Understanding and Experiences of the Process of Data Analysis

As we discussed initially, the data analysis process is fundamental to DBDM. School actors’ decision-making is therefore dependent on valid data and an analysis of good quality. Based on this, the second research question addressed school actors’ experiences and comprehension of the process of analysis associated with DBDM. Among LEA officials, this process was described as being rather unproblematic. That is, they discussed data and analysis as a completely rational process. Furthermore, they emphasized that data analysis could strengthen couplings between the LEA and schools in the sense that it offered a neutral description of schools’ results, which could be acted upon cooperatively. At the same time, certain LEA officials stressed that at times there was too much focus on data collection and not enough on the processes of analysis. This, in turn, could mean that the LEA did not get a comprehensive picture of schools’ results:

We have a lot of data today that we can use, and several are very fond of collecting data. Despite that, we do not get the whole picture … sometimes things are not backed by facts, compared to if we had used the data that we already had collected in a more qualified way (LEA Official 6).

A similar understanding of the data analysis process as a rational means of obtaining a neutral picture of schools’ results and development needs emerged among principals: “basically, data use is something very good, you don’t make decisions on loose assumptions” (Principal 12). In other words, a good analysis of relevant data can contribute to better insights into the school and to an understanding of the school’s current situation. According to principals, this insight and understanding can put focus on students’ needs, rather than teachers’ interests, when it comes to prioritizing school development areas. Data analyses can therefore reinforce a more qualified dialogue between teachers and develop a professional language. However, some principals pointed out that too much focus on standardized statistical data can discourage this positive aspect of data analysis. In practice therefore, it is, for example, important to include data that represents students’ ‘voice’ and their experiences of the teaching and instruction. Principals implied that there exists a fine line between seeing data use as an external control and as a means of professional development. As Principal 2 said, “the problem is that the LEA sometimes tend to put too much attention on pupils’ grades and results on the national tests, that sort of hard data.”

At the same time, several principals highlighted the fact that the LEA was interested in addressing school needs and development – meaning that some degree of control from the LEA was a prerequisite in the support of schools and in the allocation of resources. That is, principals did not view their relationship with the LEA as being problematic. Instead, what they perceived as being the greatest obstacles to proper data use and analysis were school organization and teachers’ skills, saying:

Obviously, we are in the middle of building up our organization around this. And we have a long way to walk … we have just started. In addition, the teachers at this school are not used to analyzing data … before I came to this school they did not have to analyze a single number (Principal 7).

Once again, strong couplings and a high degree of common perceptions emerged among interviews with LEA officials and principals regarding the analysis process linked to DBDM. Nevertheless, as we will see in the following section, couplings between the LEA/principals and the teachers are much looser. For example, teachers expressed a much more varied view and experience of the data analysis process. In addition, a more critical perspective may be distinguished among some teachers, who wonder how data analysis may benefit them or their pupils: “I am thinking, maybe we put too much time on analyzing instead of the real work to improve our teaching … maybe it is the collegial teaching planning that actually contributes to good teaching” (Teacher 22). Similarly, several teachers pointed to a central question: what was happening after the analysis?

The most difficult, but at the same time most important, work comes after the analysis. The hardest is to make real changes in the teaching practice. Absolutely, there is a risk that we put too much time and energy on data collection and data analysis. (Teacher 17)

We find the problems, but we still have not more to give, it is still the same teachers that are standing there, in the classroom, and trying to conjure up good results. (Teacher 27)

However, besides these quite skeptical attitudes and negative experiences with data use, there were also teachers who empathized with the importance of continuously analyzing data as a way of improving teaching: “I definitely think it is meaningful, there are real problems and development needs, and we can identify them with the help of data analyses” (Teacher 14).

A crucial point requiring clarification is the degree to which the data are directly linked to teachers’ own instructive practices and the students they meet with every day. Based on our results, we distinguished two different ways in which interviewees spoke of conducting analyses within DBDM. On the one hand, when teachers talked about analyzing, they primarily referred to the everyday analyses that they did in close connection with the daily teaching. That is, they pointed to different forms of ‘pulse-checks,’ in the form of tests or minor follow-ups, which they used to get insight into the relationship between their teaching and students’ learning. On the other hand, when principals and LEA officials discussed data analysis, they often referred to school-level analyses. In this case, analysis where more formal data in terms of surveys, grades, and results from different national tests were examined with the purpose to get insights about the school’s overall results. This form of structured and formal analysis requires, first, a well-functioning organization, and second, teachers with the skills and experience necessary to accomplish this kind of analysis.

The following interview excerpt may serve to illustrate this loose coupling between teachers, principals, and the LEA regarding the analysis process within DBDM:

Teacher 3:

Yes we have discussed that, the analyzing, we are not so damned good at that.

Teacher 4:

Especially when we do not have the right data …

Interviewer:

Can you give any examples?

Teacher 4:

Yes, for example, we got a spider diagram and a bar diagram and a lot of numbers. And then they told us to analyze these. First, it is very difficult to interpret these, they are difficult to compare. And after that you should compare the results with other results. How you are presenting data is very important.

Teacher 2:

… and we had a circle diagram that we should analyze, it was not so easy either.

Teacher 4:

If you instead just focus on analyzing your pupils’ achievement in relation to your own teaching, that is interesting.

Teacher 3:

Yes that is the way we use to do … previously, we only analyzed our own data linked to our teaching. In some way it is that kind of analyzing we can do. That is what we do. And that kind of analysis is fully natural, if you don’t do that, yes then you can’t develop and improve your teaching. However, the kind of analysis that we are expected to do now, that is more general and includes the whole school. And that sort of analysis requires something else …

Teacher 2:

We can’t make good analyses if we don’t understand how we should do them …

In sum, results pointed to a common approach to processes of data analysis linked to DBDM shared by principals and the LEA. Data analysis was considered primarily as a neutral, legitimate process and a tool for monitoring and school development. However, among teachers, a more varied picture of data analysis emerged. Some teachers pointed to the important implications that data analysis had for their professional development and for school development, while others expressed a more critical and skeptical attitude. However, all teachers emphasized the difficult but very important step from analyzing data to making changes in teaching practices. They meant that too much time and energy is invested in data collection and data analysis in favor of ‘concrete’ improvement in the classroom. Finally, teachers’ data analysis skills, or lack thereof, seemed to be a significant issue hindering their adoption of DBDM. Related to this, two different forms of analysis within DBDM could be distinguished in teacher interviews: informal analysis, which was closely linked to teachers’ daily teaching and formal analysis, which was more organized and focused on the school’s collective results.

Which Data?

When it came to questions surrounding which data school actors used and valued the most, differences between organizational school levels stood out. According to LEA officials, a great amount of quantitative school data is collected by the LEA, including students’ grades, results on national tests, school absence, pupils’ swimming skills, and infringement reports. All of these data were put together in a so-called supporting system for decision-making. In addition, the LEA emphasized the importance of qualitative data, such as pupil interview data or documentation from classroom observations. The latter were seen as useful, especially when it came to evaluations and analyses of specific aspects of a school’s teaching practice, such as how teachers graded or how mathematics were taught. Even if LEA officials stressed that it was important to consult varied data, they admitted to using quantitative data most often in their processes of analysis. However, the biggest challenge was not the availability of data, but rather, “a lack of systematic procedures concerning the LEA’s data analysis and how the outcomes from these analyses should be used in our decision making” (LEA Official 8).

Besides the LEA’s own data collection and internal analysis, LEA officials described how they were also highly dependent on the analyses conducted at the different schools. That is, the LEA required information and insights based on the schools’ own results and analyses. Therefore, the LEA planned which data and which questions should guide the schools’ analyses, stating simply: “we need to a greater extent choose and decide which data that should be analyzed by the schools” (LEA Official 15).

All interviewed principals’ emphasized data as an important source of knowledge when it came to better understanding and getting necessary insights into the school’s results and development needs. However, several of the principals emphasized that they did not need more data, but rather that knowing how existing data should be used. For example, in the continuous monitoring and analysis of students’ results, several principals described how they usually relied on different forms of data; this was to avoid seeking out a single result in particular, and to better understand the reasons behind results. To illustrate their use of different forms of data, several of the principals described how they tried to bring pupils’ voices into their analyses:

In addition, we collect and use other forms of data, for example, student interviews or surveys. These are soft data and I think they are important. Otherwise, the risk is that you only focus on and trust data on pupils’ achievements (Principal 2).

In general, the principals did not perceive that the LEA had too much control over or say in which data were to be used in their analyses, in particular when it comes to more qualitative data, such as classroom observation or student interviews, where they had a high degree of autonomy. However, principals often had to decide and control which data teachers would analyze, as well as which questions would be the focus of the data analysis process. This was motivated by the complexity of the analysis process in combination with the extensive amount of data that existed at the schools.

In line with principals’ experiences and perceptions, teachers emphasized the importance of using both quantitative and qualitative forms of data in processes of analysis. Students’ grades and results on national tests were, for example, seen as crucial data sources among teachers. However, due to a time-lag, this kind of data was not seen as providing great insight into current situations linked to their teaching. Accordingly, several teachers pointed out that in many cases, small tests or student interviews were more useful when it came to understanding how pupils were responding to their teaching. Compared to LEA officials and principals, teachers largely discussed and pointed to what they saw as deficits in the data. That is, that the data they were meant to analyze were sometimes incomplete or contained errors. Based on this, they described how they sometimes found it difficult to draw conclusions and decide which changes should be made. In one school in particular, the teachers described a gap between the principal’s expectations of their data analyses and what analyses were actually possible based on mediocre data. For example, they problematized the data generated by student interviews:

Are these two pupils’ opinions representative for all the students at our school? If yes, then we maybe must do some changes. Or, was it only these two who actually thought that? Or, did we formulate the questions in a way that the students didn’t understand what they actually answered? (Teacher 43)

However, even if several teachers expressed objections to the data they used for their analyses, in general they seemed to believe in data as an important knowledge source for school improvement. They did, however, point to some issues surrounding their own abilities in terms of skills and experience that were preventing them from conducting quality analyses and subsequently, translating the results of these analyses into real changes and improvements to instructional practices.

Discussion

In this final section, we will discuss the study’s central empirical findings in light of our research questions. Based on the results of how school actors make sense of DBDM, we can distinguish variety in the strength of couplings between different organizational levels. Overall, the results point to couplings between principals and the LEA as being strong in terms of the use of DBDM within the current local school system. That is, between the two organizational levels, we found a high degree of unanimity regarding the purposes of DBDM, analysis process, and which data should be defined as most important. An explanation for this common understanding can be found in the professional development program on leadership and data-based school improvement that both school system leaders and school leaders undergo over a number of years. Another reason may be the great number of forums that the LEA continuously arranges with the goal of discussing schools’ quality, results, and leadership together with principals. In these recurrent meetings, a common language and central ideas linked to DBDM seem to have been established between the two organizational levels. In contrast, we found several examples of how the couplings between these two organizational levels on the one hand and teachers on the other, is considerably more loose. Based on this conclusion, two central features of DBDM will be discussed here.

First, how school actors analyze different forms of data is important to DBDM. In our findings, we distinguished a difference between how school actors at different organizational levels understood and valued this analysis process. Teachers primarily emphasized the more informal, daily analysis in close connection to their teaching as being the most fruitful and important. On the other hand, the LEA and principals primarily emphasized formal standardized data analyses, conducted at predetermined occasions using predetermined, and often quantitative, data. Even if LEA officials and principals underlined the importance of using a diverse set of data in the analysis, it was clear that the higher we moved in the organizational hierarchy, the greater the status of quantitative data analysis. However, in line with Mausethagen et al. (2018)& Sun et al. (2016) many teachers expressed that they found this kind of analysis, which is primarily focused on standardized external data, difficult to conduct. Often they expressed that this kind of data tend to be too far from their own teaching practices. Expressed in another way, they lack what sometimes is termed as a data literacy competence (Schildkamp et al., 2014) which can be considered necessary to possess when it comes to implementing a DBDM policy. Accordingly, a large challenge to DBDM-processes is the coupling of teachers’ informal data analyses, conducted in close relation to their own teaching, with the more formal school-level analyzes. However, we did find examples of how principals managed to achieve this in part by organizing and guiding teachers’ data analyzing processes. In line with Monpas-Huber’s (2010) conclusions, this coupling seems to be a strategy of making DBDM more meaningful for teachers, which enables them to mirror and understand their students’ results in a broader school context. In addition, by getting better insight of the teaching practices at the current school, the principals also obtain a better understanding of the whole school’s results.

Second, the question, “what actually happens after the analysis?” is of prime importance. Principals and LEA officials stated many times that a central aim of DBDM was to evaluate and assess schools’ results as a means of identifying deficits and development needs – a perspective shared by teachers. However, teachers also explained how often they were already well informed and aware of the students’ knowledge gaps and the school’s deficits and development needs. The problem is that in many cases teachers felt that they did not have the capacity to handle these problems or meet the development needs. Consequently, it may be stated that teachers require not only support but also resources, in terms of time and skills, in their implementation of continued improvement processes. In sum, this points to an important risk associated with DBDM policy, which is that too much focus is put on the data and on the process of analysis. That is, DBDM has become too ‘back-heavy’ in that too much energy and time are spent on collecting and analyzing data for the purposes of identifying weakness in pupils’ knowledge and in teachers’ approaches, and not enough time is being dedicated to innovation and improvement. In line with Cosner (2011), DBDM was found to be an important component of efficient school improvement, but not the only one. In addition to a well-functioning organization and school actors that are able to analyze different forms of school data, teachers’ subject matter knowledge seems to be crucial when it comes to making use of the information that comes of the analysis to create innovative instructional practices.

Implication of the Study

Based on the results and the discussion above, two main implications for leadership and policy linked to DBDM can be distinguished. Firstly, principals and LEA officials must be aware of the varied understanding of DBDM that many times exist between school actors’, at different organizational levels. That is, if the teachers, for example, primarily see DBDM as a mean of external control and monitoring, and not as a powerful strategy to reinforce professional development and improving the teaching practice, it will be difficult to obtain legitimacy for a DBDM policy.

Secondly, when it comes to organizing and leading DBDM processes principals must consider the difference between the so-called informal and formal processes of data analysis. If teachers, on the one hand, primarily find the informal analysis, that they conduct in close connection to the everyday teaching, as most valuable and meaningful and the principals, on the other hand, in greater extent focus on and find the formal school-level analyses as more important, it may result in the couplings between these organizational levels becoming loose, which in turn can become an obstacle for an effective DBDM at the school. Consequently, a crucial factor for a successful DBDM is that school leaders must be aware of the difference between the informal and formal analysis and organize and lead the data analysis processes in a way that couple them tighter together.

Limitation of the Study

Due to the qualitative approach of the study, the conventional way of generalization is limited. However, based on the study’s methodological design, with a consistent theoretical framework and previous research we although find it possible to contribute to analytical generalizations, which in turn has generated important results and insights to the research field and new questions for further investigations (cf. Kvale & Brinkman, 2014). However, the municipality and the LEA in focus for the study is, from a Swedish perspective, quite large compared to other Swedish municipalities. In light of this, it would be interesting to make further investigations with similar approach but in a smaller municipality. Such a study may illuminate whether the size of the local school system and the LEA have impact on school actors’ experiences and understanding of DBDM.

Disclosure Statement

No potential conflict of interest was reported by the author(s).