Investigative modes in research on data use in education

ABSTRACT

This article reports on the findings from a literature review of research on data use in education published in English, German and Scandinavian languages. The review is inspired by methods for systematic mapping. The analysis illustrates how the characteristics of the total corpus of 129 articles on data use in education vary across different contexts, countries and regions. In all contexts, the studies primarily investigate structures and systems around data use. While the Anglophone studies are mainly empirical and often concerned with implementation and effectiveness in terms of data use, the studies published in German and Scandinavian languages focus more heavily on discussions and analytical reflections upon the developments of data use in education. Six investigative modes of studies on data use that can contribute to creating a more nuanced understanding of research on data use in education are identified, presented and discussed.

KEYWORDS:

• Data use
• investigative modes
• evidence
• systematic mapping
• literature review

Introduction

This article reports on the findings from a literature review of research on data use in education. The review method is inspired by methods for systematic mapping and narrative analysis. Although data use has been studied, particularly in the USA, we know less about research on this topic and how data use is investigated in different contexts, countries and regions. In this article, we focus on data from student achievement tests produced for external evaluation and monitoring expected to be used by teachers, school leaders and district administrations for the purposes of school and professional development.

Data use represents the centrepiece of what is often referred to as evidence-based governing regimes, which are considered an ideal way to coordinate activities on different levels in a school system (Ozga, 2009). A range of countries have implemented a relatively new set of governing approaches that emphasize the combined power of performance measurement and hands-on management. Using quality indicators, goal setting, result monitoring, accountability and competition, local authorities, school leaders and teachers in most of the modern world are expected to initiate concrete actions to improve student achievement (Altrichter & Merki, 2010; Fuller, 2008; Gunter, Hall, Serpieri, & Grimaldi, 2016). This development can be considered as part of an overarching and contemporary global policy message communicated through similar concepts (e.g. quality, accountability, learning outcomes, evidence based, formative assessment and decentralization), and tends to lead to developments in education directed towards achieving the same and seemingly universal goals (Karseth & Sivesind, 2010; Pettersson, Prøitz, & Forsberg, 2017; Røvik, 2014). Yet local, regional and national variations more often characterize the field of education (Hall, Grimaldi, Gunther, Møller, Serpieri & Skedsmo 2015; Hopmann, 2008, 2015). For example, accountability in the USA is described as high-stakes owing to use of incentives and sanctions related to student performance data, while European and Scandinavian countries are often recognized as low-stakes and ‘half way accountability’ or ‘soft accountability’ owing to a lack of such incentives and sanctions (Hatch, 2013; Easley & Tulowitzki, 2016). Moreover, it can be argued that manifestations of decentralization and agency differ between the Scandinavian countries and countries in Northern Europe and the USA (Møller & Skedsmo, 2013; Gunter et al., 2016; Prøitz & Aasen, 2017), and traditions for assessment and grading even vary considerably across the Scandinavian countries (Lundahl, Hulten, & Tveit, 2017). Variations in curriculum development between countries have been described as reflecting long-standing traditions in approaches to schooling in terms of curriculum versus didaktik (Hopmann, 2015; Lundgren, 2006; Karseth & Sivesind, 2010). Moreover, what is considered as evidence in education has been observed as differing between contexts, regions and countries (Gough, Tripney, Kenny, & Buk-Berge, 2011; Rieper & Hansen, 2007). Consequently, it is argued here that differences in education between contexts, regions and countries can be understood as reflections of varying traditions and values of schooling.

With these examples in mind, a focal point for this literature review is whether research on data use in education, as a central part of the contemporary evidence movement in education, reflects such differences. As such, this article presents a study of another overarching and trending concept in education through an investigation of how data use in education is understood in research in different geographical regions and countries.

The aim of the study is to investigate the main characteristics and results of existing research on data use in education published in English, German and Scandinavian languages, and to identify distinct and typical features of this body of research across the geographical regions these languages represent. By including studies in different languages, it is assumed that similarities and variations in research on data use will enable comparison of research from different geographical regions.

The following research questions were developed to guide the study: What are the main characteristics and results of existing research on data use in education? What characterizes the approaches in research on data use published in English, German and Scandinavian languages? The article is organized in four sections. First, an outline of the scope of the review and the data use theme is presented. Secondly, the method and selection process for the review study is described. In the third and fourth sections, the results of the literature review are presented and discussed.

Scope of the review

The scope of this review study is based on studies on data use in education. The more substantial Anglo-American knowledge base, in particular, provided a starting point for the chosen review approach: it enabled the identification of several focus points, constructs, logics and concepts deemed important for the definition of the research questions and the scope of this review.

Within the field of data use in education, there are few existing informative literature reviews. Coburn and Turner (2011) find how the ways in which data are used depend on organizational factors such as access to data, time, norms of interaction and leadership. Little (2012) discusses how data use typically requires translation into more practical knowledge. She argues for more extensive use of micro-process studies to enhance knowledge about how local practices both construct and instantiate organizational routines and processes. While these reviews take a more conceptual and methodological approach, Sun, Przybylski, and Johnson (2016) are concerned with ‘best practices’ around school leaders’ data use. They are particularly concerned with teachers’ low competence and efficacy in using student data to inform instruction. Based on the review, they suggest ways that school leaders could work ‘evidence-based’ to develop teacher competence in using data to inform their classroom instruction. Also taking the implementation of data-based decision making as a starting point in the quest to improve student learning, a systematic literature review conducted by Hoogland et al. (2016) highlights teacher collaboration around the use of data, data literacy and leadership as a necessary prerequisite for successful data use in the classroom. Based on their review on studies on data use and school leaders, Hornskov, Bjerg, and Høvsgaard (2015) argue that data use should be seen in a process perspective that is sensitive to school context, also capacity building needs to be customized to suit the purpose of data use on the different institutional levels. Existing literature reviews have been conducted with a specific actor, group or context in mind and with a primary focus on studies published in English. This study builds on and supplements these contributions by focusing on research published in several languages, to include research from German and Scandinavian countries.

Review approach

This literature review draws on well-known procedures defined in the literature on research synthesis (Davies, 2000; Thomas & Pring, 2004; Gough, Oliver, & Thomas, 2012). More specifically, the current study shares similarities with what is often characterized as a systematic mapping and narrative analysis in which different elements derived from individual studies are identified and configured into a new, macro-conceptual and/or theoretical understanding (Gough et al., 2012). As the aim of this study is to explore a heterogeneous topic in diverse literature, investigated using various research designs and involving different groups of actors for an analysis and discussion of the investigative modes of the field, this review can also be viewed as sharing methodological ideas with the meta-narratives developed by Greenhalgh et al. (2005). By focusing on the main characteristics and results of research on data use, the distinct and typical features of this body of research are studied. The systematic mapping approach applied provides an overview of the characteristics of the research field in terms of several descriptive categories and codes (see Appendix 4).

An effort has been made to broadly cover the research field by examining varied approaches to the study of data use in education; this entails the inclusion of heterogeneous primary studies and involves interpretive analyses (Gough et al., 2012). An overarching review protocol was developed to ensure the application of the general principles of transparency and replicability of the research synthesis (Gough, Oliver, & Thomas, 2013; Hammerstrøm, Wade, & Jørgensen, 2010). The review protocol aims for concept saturation by focusing on the identification of a range of concepts and theoretical frameworks (see Appendices 1 and 2).

Searches for academic, international, peer-reviewed and scientific published articles were performed using the Education Resources Information Center (ERIC) and Scandinavian and European databases. The initial scoping for studies in several databases (such as Web of Science and Scopus) proved ERIC/EBSCO to be the most efficient source for the purposes of identifying a broad range of approaches to data use in education while aiming for saturation. Searches for sources in the Scandinavian languages were conducted in the Norart, Idunn, BibliotekDK, Artikelsök and DIVA databases, and in the German-language database Fachinformationssystem Bildung. However, since there is no common Scandinavian database that enables searches for only peer-reviewed publications, a pragmatic choice was made to search for literature broadly characterized as scientific publications, and consequently, the initial searches of Scandinavian databases had a wider scope than the English- and German-language studies. This search strategy was used to identify the studies that were most relevant to the literature review. Only peer-reviewed articles were included.

The initial scoping for relevant studies in the different databases revealed that the concept of data use is framed differently in different contexts and in different languages. To ensure the best possible equivalence of the search queries in English, German and the Scandinavian languages, a set of descriptors and keywords was developed. All descriptors and keywords used were chosen based on consultations with experienced researchers in the field of education in each of the relevant languages, and a range of search queries for the five chosen languages was developed and applied (see Appendices 1 and 2). A set of inclusion and exclusion criteria was developed in accordance with the scope of the study, and a three-phase screening process was applied (see Appendix 3 for the inclusion criteria and screening process).

Coding and categories

The analysis was performed in three steps. The first step involved reading the included publications, and by doing so obtaining an overview of the central research foci or themes of the studies. In the second step, the publications were coded and categorized by a coding scheme with various categories (see Appendix 4 for an overview of the categories and codes). The coding scheme developed for this study was inspired by the work of Wilson (2014) and Gough et al. (2012) and previous work on systematizing and analysing research in education (Aasen, Prøitz, & Borgen 2005; Aasen & Prøitz, 2003, 2004; Larsen & Prøitz, 2005; Prøitz, 2010). The coding of the identified studies used several descriptive variables: year of publication, first author’s country of origin, language, school level focus, actor focus, research question and aim of the study, type of study, method of study and results of the study. The variables chosen for this study were selected based on their relevance to the research questions in terms of descriptively identifying the characteristics and trends in research on data use and identifying the approaches of the data use studies. This provided the study with descriptive information that could be quantified and combined for an overview of the main characteristics of the included studies. In the third step, the paragraphs defining and describing the study’s research questions and/or aims, data and selection and results were identified and extracted to facilitate the qualitative identification and interpretation of patterns. The interpretive approach used in this step of the review was inspired by Noblit and Hare’s (1988) work on lines-of-argument synthesis, and more recent work using core studies and key concepts as structuring and configuring elements (Lillejord & Børte, 2016). The extracts were configured through comparison of patterns of similarities and differences and clustering of meaning within and between categories.

Analysis of the total literature corpus

In the following section, the results of the review are presented in two stages. First, an overall analysis of the central characteristics of the total data use literature corpus identified is presented. Secondly, a more detailed analysis of the publications within the three language groups studied (English, German and Scandinavian) is described and discussed.

The 129 studies included in this review spanned a period of 15 years. The frequencies of the publications fluctuated substantially according to the year. For example, in some years, there was a considerable number of publications (such as in 2006 and 2012), while in other years, there were few (such as in 2001 and 2005) (Figure 1).

Figure 1. The 15-year range of included publications on data use, 2000–2014 (N = 129).

The span of publications per year and per language shows that publications in the different languages peak during different years: English in 2006, 2010 and 2012; German in 2002, 2006, 2007, 2008 and 2013; and Scandinavian languages in 2006. Figure 1 also shows that more than half of the English (34/56) and German (34/54), and more than one-quarter of the Scandinavian (5/19) studies were published during the peak years of each of the language groups. The peaks can be an indication of sudden events in the education field being heavily studied and/or debated, or they can be reflections of a special issue in a particular journal that emphasized a certain theme. They can also be indications of specific attention being given to a certain topic, perhaps as a result of shifts in political priorities and increased research funding.

Overall characteristics of the total literature corpus

Several characteristics describe the total corpus of the data use literature included in this study. As mentioned, the studies cover a period of 15 years, in which the peak years were 2006–2007 and 2012–2013. A general and rising trend of studies on data use in education has also been observed. The countries of origin of the studies’ first authors show that the publications emanate from a range of countries. The two leading countries are the USA and Germany, and slightly more than 50% of all the authors are European.

Few of the studies included explicitly state the school level focus; one-third applied a ‘whole-school’ approach, while another third indicated a focus on primary and secondary school. Likewise, most of the studies examine a broad range of actors. Teachers are the dominant single group of actors examined, while students are the least studied single group of actors. In one-third of the studies, there is a combined focus on several groups of actors (teachers, school leaders and local authorities). Several studies focus on the relations between teachers and school leaders and other actors in schools. There are fewer studies that investigate the relations between the school level and the local district/authority level. This reveals that the literature corpus is strongly focused on structural changes and the use of tools to develop and improve practices at the school level and within schools. The extent to which teachers integrate data use in their professional discussions about teaching practices and student learning activities appears to be minimal. This indicates that there are few studies on how data are used by teachers. Another one-third of the studies do not explicitly state the actor of focus, and these studies are among the one-third of theoretical/analytical studies identified in the total literature corpus. These studies often investigate more general issues concerning data use in education. Most studies are empirical, meaning that they are mainly qualitative studies and interview-based studies. There are few quantitative studies represented, and only 1% of the studies purport to study the effects of data use or are intervention-based studies.

Studies published in English

Studies published in English are mainly written by American authors (c.f. Coburn & Turner, 2011; Little, 2012; Wayman, Jimerson, & Cho, 2012). Other Anglophone countries are represented by authors from the UK, Australia, Canada and New Zealand (c.f. Anderson, Leithwood, & Strauss, 2010; Chick & Pierce, 2012; Goldstein, 2001; Kelly & Downey, 2011; Robinson, Phillips, & Timperley, 2002), and some studies have first authors originating from the Netherlands, Italy and Belgium (c.f. Paletta, 2011; Schildkamp & Kuiper, 2010; Vanhoof, Verhaeghe, Verhaeghe, Valcke, & Van Petegem, 2011).

A substantial proportion of the studies are published in journals that focus on leadership and school effectiveness (e.g. School Effectiveness and School Improvement and Leadership in Policy and Schools). Another considerable proportion of the studies are published in journals with a more general educational scope (e.g. American Journal of Education and Teachers College Record). A few studies are published in more thematic-oriented journals (e.g. Assessment in Education, Educational Psychologist and Mathematics Teacher Education and Development).

One characteristic of the studies is that they are mainly empirical studies with an investigative focus on the whole school, as often the studies focus on more than one actor, mainly teachers in combination with school leaders, and school leaders and local authorities (c.f. Datnow, Park, & Kennedy-Lewis, 2012). Teachers are the most researched actor group, while students are the least studied actor group, researched in only two studies (Murnane, Sharkey, & Boudett, 2005; Supovitz, 2012).

To carry out a more detailed explanation of the scope and results of these studies, their research questions and aims, as well as discussions of the results and conclusions, were mapped and categorized. Two primary investigative scopes of the investigation of data use in education have been identified. The first is represented by studies that investigate how data come into play as seen from a predominantly top–down/implementation perspective, characterized by their focus on the introduction and/or implementation of different types of results-oriented systems, data, models or ideas about data use. Often, these studies focus on the challenges, affordances and constraints observed and the experiences of the actors at various levels of the education system – particularly teachers and school leaders. Examples of research questions or aims of the studies in this category of investigative scope are:

How do different implementation cultures come about regarding data use? What is the role of leaders in shaping these different orientations? (Park, Daly, & Guerra, 2013)

How does DDDM, with its technical–rational theory of action, play out at the local level as teachers attempt to use data? What is the context for teachers’ use of data at the district, school and department levels? How do teachers use data and what instructional changes result from DDDM policy? (Datnow et al., 2012)

The other investigative scope identified approaches the issue from a more explorative and bottom–up perspective, often in terms of sense-making and micro-practices in schools. This approach can be exemplified by the following research questions:

For which purposes are different types of data being used by school leaders and teachers in secondary education in the Netherlands? Which variables promote or hinder data use by teachers and school leaders? (Schildkamp & Kuiper, 2010)

The purpose of the inquiry was to examine the nature and benefits of systematic inquiry in schools and to understand the forces that serve to enhance or impede schools’ propensity to embrace evaluative inquiry as a support for decision making. (Cousins, Goh, & Clark, 2006)

In terms of results, the studies’ discussions of the opportunities and challenges related to data use are prominent (c.f. Kelly & Downey, 2011). The studies argue that data are more often used for documentation and control purposes than for development. A substantial proportion of the studies’ results indicate different barriers to and difficulties in engaging teachers, and often explore ways to motivate teachers. An important part of the results presented in many of the studies points to and/or discusses aspects of teachers’ lack of competence in using data (Dunn, Airola, Lo, & Garrison, 2013; Klenowski & Wyatt-Smith, 2012; Parr & Timperley, 2008). Few studies address the more complex issues regarding why this is a challenge.

In almost all of the 56 studies published in English, the role of the school leader is found to be of great importance, especially with regard to solving problems through data use (Little, 2012; Schildkamp & Kuiper, 2010). This key role of school leaders is often observed in relation to the priorities, strategies and policies of districts and their views on data use. The results of studies on districts and local school authorities highlight how variations in schools’ efforts towards data use seem to reflect districts’ emphases on data use, in terms of whether there is a policy on data use, whether it is prioritized or whether it is a supported or even a coerced activity by local school authorities (Kerr, Marsh, Ikemoto, Darilek, & Barney, 2006; Miller, 2010). Within this line of enquiry, an important aspect for the development of data use in schools is how different organizational cultures frame, foster or hinder data use in schools, such as how a high degree of coerciveness linked to control seems to weaken the development of productive organizational cultures of data use (Coburn & Turner, 2011; Wayman et al., 2012; Young, 2006; Young & Kim, 2010). Several studies demonstrate that teachers seem to take data into consideration by using an eclectic and intuitive approach. The research identifies teacher competence, school leadership, organizational culture and local authority involvement as important factors concerning what fosters or hinders data use. The general concern of these studies involves questions regarding what should be done, and how to overcome problems related to a lack of teacher engagement in data use.

Studies published in German

Most of the studies published in German have a German first author, but authors from Switzerland, the Netherlands and Austria (c.f. Husfeldt, 2004; Moser, 2013; Schratz, 2013) are also represented. A substantial number of the studies are published in journals with a special issue on data use, such as the Empirische Pädagogik (2007), Journal für Schulentwicklung (2013) and Bildung und Erziehung (2002). The special issues focus on data use in relation to standardized testing, approaches to and discussions of what works in school development, and studies related to the existence, use and challenges of various types of data, and results from evaluations.

In contrast to the studies in English, the majority of these studies do not investigate data use in relation to particular education levels or in relation to certain groups of actors, although some studies adopt a whole-school approach with a special focus on teachers. Most of the studies are either theoretical and analytical studies, discussion papers (Beywl, 2013; Bos & Schwippert, 2002; Böttcher & Keune, 2013; Klieme & Stanat, 2002; Van Ackeren, 2002), review studies (Breiter & Stauke, 2007; Büchter & Leuders, 2005; Isaac, Halt, Hosenfeld, Helmke, & Gross Ophoff, 2006) or survey studies (Diedrich & Pietsch, 2013; Maier, 2008; Wacker & Kramer, 2012). This is in stark contrast to the studies in the English language, which are all empirical studies or review articles of empirical studies.

The investigative scope of these studies can be defined into two broadly conceptualized categories, but of another kind than for the studies in English. In the first category, studies that provide an overview of the field, various types of evaluations and tests or review studies in various contexts were identified. Examples of how the scope is described are as follows:

Ein Überblick über Forschung zur Nutzung von Leistungstests [Overview of research on use of data from student performance tests]. (Bonsen & Frey, 2013)

Beschreibung der aktuellen Praxis von externer und interner Evaluation im Rahmen des Steuerungsparadigmas [Description of current external and internal evaluation from a governance perspective]. (Gärtner, 2013)

The other investigative scope identified is characterized by studies with a specific focus on what data and data-based decision making or output-oriented governing can offer, and what kind of opportunities for improvements the data can support, as well as possible limitations of data use. The following examples illustrate research questions and aims from this category of studies:

Möglichkeiten und Grenzen von internationalen Schulleistungsuntersuchungen [Opportunities and limitations in using international comparative performance tests]. (Bos & Schwippert, 2002)

Inwiefern können Outputmessungen zur Unterrichtsqualität beitragen? [To what extent can output data contribute to enhancing the quality of teaching?]. (Van Weeren, 2007)

The few empirical studies that are mainly based on various types of survey data produced similar results to the studies written in English (i.e. the importance of teacher competence and school leadership for productive data use). These studies investigate teachers’ and school leaders’ perceptions and experiences with a certain standardized test. In some of these studies, different types of data use are presented, such as for internal use, for quality development processes and for external communication. The latter are noted to have been more recently established (Nachtigall & Hellrung, 2013). In contrast to the studies in English, data use for control is more explicitly articulated to be detrimental to development processes. Thus, it could be argued that these studies present a more critical voice than the studies in English. This critical aspect is prominent in the conclusions of the theoretical articles; however, a substantial number of these studies also discuss the possibilities that data offer for school development. In a variety of ways, these studies are indicative of a research field that reflects more general issues concerning questions related to both advocating and opposing recent developments in continental European education.

Studies published in Scandinavian languages

The majority of the Scandinavian studies are written by Norwegian and Swedish authors in their native languages. A substantial proportion of the studies identified by searches in the Nordic as well as the individual country databases was excluded owing to the lack of focus on data use. Each of the 70 initially identified Scandinavian studies touches upon concepts such as data, data use, tests, grades, test results and national tests; however, these concepts are mostly used to debate issues related to other aspects of schooling, such as teaching and learning processes and assessment in a broader sense, such as new policy developments. Data and data use are used as contextual frames for other themes under investigation. Several of the excluded and included studies reflect a tendency to debate the introduction of a stronger, results-oriented and accountability script in education in the Scandinavian countries in general. In Norway and Denmark, this debate revolved around the introduction of national tests in the early and mid-2000s, and in Sweden it concerned the earlier introduction of grades in primary school (c.f. Bjerre, 2009; Fasting, 2006; Forsberg & Wallin, 2005; Haugstveit, 2005; Jönsson & Thornberg, 2015; Lundqvist & Lidar, 2013; Lunneblad & Carlsson, 2010; Moos, 2006; Skaftun, Solheim, Roe, & Narvhus, 2006).

The studies written in Scandinavian languages are mainly published in scientific journals with a strong pedagogical perspective, and most are found in Norsk pedagogisk tidsskrift, Pedagogisk forskning i Sverige, Nordisk pedagogik, Nordisk pedagogisk tidsskrift and Nordic Studies in Education. It should be noted that four of the 19 Scandinavian publications were published in 2006 in a Norwegian journal (Norsk pedagogisk tidsskrift) in a special issue on the reintroduction of national testing in Norway.

The studies can be divided into two main groups. The first includes studies that investigate and discuss methods and the reasoning and constructs behind various national tests introduced in Norway at the beginning of the 2000s and during the mid-2000s, along with their usefulness for teachers and school leaders. The second includes studies that debate different approaches to tests, data and data use from various perspectives, such as sociological perspectives of categorization and a local school authority perspective, the usefulness of using test data in education from an evaluative education reform perspective and the usefulness of using data to evaluate the effects of reform. These studies adopt a whole-system approach and question how teachers and school leaders might use the data provided, or their functionality in practice. Most of the studies are document analyses, review studies and qualitative studies using interview and observation methods. Few studies are based on quantitative data, although a couple of the studies refer to statistical information.

The investigative scope of the first group of studies focuses on methodological issues and construct-related issues for the various national tests:

Artikkelen presenterer og drøfter den modellen som dannet grunnlaget for nasjonale prøver i skriving, og dermed implisitt hvorvidt det er mulig å imøtekomme en slik flerfasettert intensjon ved vurdering av elevers skrivekompetanse [The article presents and discusses the model for the national test in writing, and thereby implicitly questions whether it is possible to accomplish a multifaceted intention inherent in the assessment of student competence in writing]. (Fasting, 2006)

I denna artikel kommer ett test som används i den svenska grundskolans tidigare år att diskuteras, lästestet »Vilken bild är rätt?» (Lundberg, 2001). Testet kommer att jämföras med Skolverkets språkbedömningsverktyg »Nya språket lyfter» (Skolverket, 2008). I artikeln kommer testen att relateras till frågor om makt, etik och demokrati. En central fråga kommer att vara: Vem har makt över bedömningen och med vilken legitimitet? [In this article, a test for reading proficiency that is in use in the Swedish primary school will be discussed. The test will be compared using Skolverkets’ linguistic proficiency assessment tool. The article discusses these issues in relation to questions about power, ethics and democracy. A central question is: Who has power over assessment and with what legitimacy?]. (Wedin, 2010)

The other group of studies adopts a varied set of approaches that focus on several aspects related to a stronger result orientation in education:

Har elevenes læringssituasjon endret seg etter innføringen av Kunnskapsløftet (K06)? Eller, mer konkret, har reformen ført til en bedre kultur for læring og bedre læringsutbytte for elevene? [Have students’ learning situations changed since the introduction of the knowledge promotion reform? Or, more specifically, has the reform led to an improvement in the learning environment and the student learning outcomes?]. (Olsen, Hopfenbeck, & Lillejord, 2013)

For two decades now, the state, its management and institutions seem to be changing this dominant discourse of the public and political arena into a discourse of effectiveness adhering to the market place or to the administrative and management arena. In this article, this transformation is discussed and so are the – not often discussed – administrative technologies that lead to the new situation. (Moos, 2006)

Artikeln utforskar utvecklingen av nationella bedömningsinstrument, så som nationella prov, i svensk och norsk grund- och gymnasieskola. [This article investigates the development of national instruments for assessment, or example national tests, in Swedish and Norwegian elementary and secondary schools]. (Lundahl & Tveit, 2015)

The results of the studies often led to calls for caution when working with data. This is interesting because few reported on studies of data use in practice. Most of the studies are discussion papers and analyses of theoretical perspectives, or policy developments discussing the broader aspects of the education system, and, to a certain extent, they raise questions more often than they present results. Some of the central issues raised, similar to the more critical studies in German, are questions regarding the effects on teachers as professionals in a more results-oriented system. The Norwegian and Danish studies indicate a certain ‘newness’ of the field of data use, which seems to be reflected in the few empirical studies regarding what teachers do with data in schools.

Six investigative modes on data use in education

The analysis illustrates differences between research fields on data use in education. To a large extent, the studies in English define the main characteristic of the field based on empirical results, while the studies in German and Scandinavian discuss and question issues related to the mere existence and application of data use in education. Consequently, the studies in German and the Scandinavian languages involve a more diverse and broader understanding of data use that is seldom related to a particular education level or specific actors, although teachers and school leaders are highlighted in several of the studies. The studies written in the German language were published in scientific journals with more explicit didactical and school development approaches than the studies published in the English language. The studies written in Scandinavian languages were mostly published in pedagogy-oriented journals. Only a few of the German- and Scandinavian-language studies are empirical, whereas the majority of the English-language studies are. The studies in English seem to be more concerned with the larger and systemic developments, positioning the school actors, such as teachers and school leaders, as executors within education systems driven by data. They also focus on questions regarding the degree of compliance with the structures and pressures of the education system as consequences of the increased access to data and expectations of data use.

Based on the analysis, six broadly defined investigative modes (overlapping and non-mutually exclusive) of studies on data use in education have been identified and defined: (1) implementation studies; (2) explorative studies; (3) overview studies; (4) discussion studies; (5) methodological studies; and (6) system-critical studies. The first three modes represent the largest groups of studies, while the final three modes represent smaller, yet still significant, groups of studies. The two investigative modes of ‘implementation’ and ‘explorative’ studies are mainly Anglo-American studies that focus on the whole school system and are mainly based on qualitative empirical data. Most of these studies are concerned with improving data use in schools, and thus educational quality and learning outcomes, by identifying the factors that would positively contribute to this development. The investigative modes of ‘overview’ and ‘discussion’ studies are most often observed among studies written in German. These tend to be mostly theoretical overviews and review studies that discuss opportunities for data use in school development, on the one hand, and criticize the development on the other, which together appear to provide a combination of perspectives in a nuanced debate. The investigative modes of ‘methodological’ and ‘system-critical’ studies are mostly found within the Scandinavian-language studies. They are characterized by a certain ‘newness’ of the field, illustrated by studies that discuss the quality and requirements of tests and tools for data selection and data handling on one hand, and examine and question the consequences of new education policies for schools, teachers and, sometimes, students on the other. Following from educational accountability and data use, some of these studies are concerned with the changes in teachers’ work and the teaching profession.

A relatively large number of the ‘implementation’ studies can be classified as ‘school effectiveness’ literature. This implies that the studies are concerned with ‘what works’ and how to overcome barriers to use data productively. These studies also typically adopt a top–down perspective with a special focus on leadership at the district/municipal and school levels (i.e. a whole-school approach). Within this literature, teachers are primarily considered to be implementers, while school leaders are considered to play key roles in solving issues related to data use. This body of research is different from the ‘explorative’ studies, the ‘discussion’ studies and the ‘system-critical’ studies, which often examine more didactical issues in school development and teacher professionalism, and place teachers in more autonomous positions. The ‘discussion’ studies and the ‘system-critical’ studies also place a stronger focus on the potential social effects and impacts of new results- and performance-oriented education policies through the introduction of new tests and in relation to teachers’ grading and assessment practices. Some of the studies use methods that are more oriented towards dialogues and interactions between teachers and students through the use of ethnographic and phenomenological approaches, often in order to scrutinize contemporary education policy.

Discussion and conclusion

In this study, we questioned the main characteristics and results of existing research on data use in education and examined what characterizes the approaches in research on data use published in English, German and Scandinavian languages. The study reveals evidence of a general increase in the number of studies on data use in the period studied. However, the studies also reflect a European unfamiliarity with student performance data and data use in education that is not observed in the Anglophone studies, which is interpreted as reflecting variations in the degree and history of results orientation in the various education systems (e.g. Easley & Tulowitzki, 2016; Gunter et al., 2016). Across all contexts, the included corpus of studies investigates structures, systems and the introduction of these systems. The studies also illustrate the experiences and effects of various programmes for data use. The review has identified a lack of research on the actual data use practices of teachers in general. Teachers are the single most often studied actor, but they are more often viewed as one among many other system actors than the main actor of interest. Another important actor that can be considered understudied is the student: none of the 129 studies solely investigated data use and students. This literature study shows how this research topic seems to be more oriented towards the structures of governance and management than the processes of teaching and learning. Consequently, this could explain the lack of research on the core activities of the prime concept that data use is meant to support.

Another observation that can be considered related to variations in traditions of schooling as well as variations in understandings of data and evidence is the substantial portion of the reviewed literature that focuses on ‘what works’ to increase academic achievement through forms of evidence-based practices as part of data use practices. In the Anglo-American literature in particular, there is less emphasis placed on more holistic approaches to plan, carry out and evaluate teaching in ways that necessitate the involvement of teachers. In the studies written in German and Scandinavian languages, less of a focus is placed on empirical studies of how teachers plan, carry out and evaluate teaching. This issue is interpreted as related to how attention to student outcomes and external evaluations are rather new developments, and related to where tensions are created as a result of changes in how teachers and school leaders are positioned as key actors and their autonomy is challenged (Mausethagen & Smeby, 2016; Mølstad & Karseth, 2016).

This finding can be understood as a reflection of two long-standing and divergent traditions of governing schools according to the curriculum traditions of the USA and English-speaking countries, and the tradition of ‘didaktik’ in German-speaking countries, as well as in Scandinavian countries (Hopmann, 2008, 2015). The studies emanating from a continental European tradition mainly focus on data use as it is integrated into internal evaluation procedures, where the teaching profession has a relatively high degree of ownership over the data produced. Studies from the USA mainly focus on how data from standardized tests are used in relation to external evaluation processes, and consider teachers’ didactic practices less often. Based on these broader traditions, the six investigative modes identified in this study offer a more refined and nuanced understanding of approaches in research on data use in education.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This work was supported by the Norges Forskningsråd.

Notes

1. The high number of hits reflects the wider search strategy for the Scandinavian language publications than for the English- and German-language publications. The high number of hits includes articles in newspapers and popular magazines (Artikelsök), among other publications, and a substantial number of duplicates in the two Nordic databases Norart and Idunn.

Appendix 1.

Overview of descriptors and keywords used in searches in five languages

Table

Language	Descriptors	Keywords
English	Data use	AND: superintendents, principals, school leaders, teachers, students, primary and secondary school, primary school, secondary school, middle school, grammar school, comprehensive school, organisational development, professional development, government, local authority, regional authority, municipality, school district, district officials, local officials
	Evidence
German	Daten/Daten Nutzung	AND: schulentwicklung, evaluation
		Swiss: Primarstufe: Primarschule Sekundarstufe I: Realschule, Sekundarschule, Bezirksschule, Oberstufenschule
		German: Primarstufe: Grundschule Sekundarstufe I: Hauptschule, Realschule, Gymnasium, Gesamtschule (sowie neu geschaffene Schulformen: Regionalschule, Erweiterte Realschule, Realschule plus, Oberschule, Regelschule, Sekundarschule, Staddeilschule)
		Austrian: Primarstufe: Volksschule Sekundarbildung Unterstufe: Hauptschule, AHS-Unterstufe, Koopera<ve Midelschule, Neue Midelschule
Norwegian	Bruk av elevresultater, karakterer, prøveresultater, nasjonale prøver, elevdata, skoledata, evidens	AND: skoleleder, rektor, lærer, elev, kommune, fylkeskommune, kommunesektor, lokalt myndighetsnivå, lokale myndigheter, ungdomsskole, grunnopplæring, skoleutvikling, organisasjonsutvikling, profesjonsutvikling, kompetanseheving
Swedish	Användning av läranderesultat, elevresultat, studerande framgång, betyg, provresultat, nationella prov, elevdata, skolresultat, bevis	AND: skolledare, rektor, lärare, elev, kommun, landsting, kommunal sektor, kommunal nivå, lokala myndigheter, skola, grundskola och gymnasium, skolutveckling, professionell utveckling, organisationsutveckling, kompetensutveckling, kompetensheving
Danish	Brug af elevresultater; karakterer, afgangsprøver; nationale test; elevdata, skoledata; evidens	AND: skoleleder, lærer, elev, kommune, kommunale sektor, forvaltning, lokale myndigheder, skole, folkeskole, grundskole, skoleudvikling, organisationsudvikling, professionel udvikling; kompetenceudvikling, pædagogisk udvikling

Appendix 2.

Search queries in German language

Table

Tool/method	Purpose of use	Data/results/effects
Schulinspektion	Unterrichtsentwicklung	Nutzung
Schulevaluation	Qualitätsentwicklung	Noten
Lehrerevaluation	Organisationsentwicklung	Schülerergebnisse
Schülerbeurteilung	Schulqualität	Indikatoren
Beurteilung	Massnahmen (Umsetzung)	Schulleistung
Bewertung	Qualitätsmanagement	Wirkung
Assessment	Governance/Steuerung/ Handlungskoordination	Wirksamkeit
Bildungsstandards	Bildungspolitik	Evidenz
Controlling		Befunde
Monitoring

Appendix 3.

Inclusion criteria and screening process

An overarching criterion for the literature study was to include studies of data use that focus on data use practices, which delimits the focus of the study to the local/district authority level and the school leadership and school level. The overarching inclusion criteria for the review were: the five languages (English, German, Norwegian, Danish and Swedish), the 2000–2014 period, primary and secondary education, and assessment and evaluation studies for school development and governance purposes. More specifically, this refers to studies that take an empiric, analytic and/or theoretical approach to studies of data use in primary and secondary education for organizational development or professional development.

Furthermore, studies with a general focus on data use at the school level and the local authority/district level, as well as between local and central authority levels, and in varied school subjects were included. As the review focused on the characteristics of research on data use in education, studies that investigated data use in special education, or that concerned particular or marginalized groups, were excluded. The focus of the review was also delimited by an understanding of data primarily as the results of formal assessments.

The inclusion and exclusion process was divided into three phases of screening: (1) title and keywords; (2) abstracts; and (3) full-text publications. A coding manual was used to categorize the full-text publications included (see Appendix 4 for categories and codes). Phases 1 and 2 were conducted by a team of researchers. During phase 3, a double coding process was used for a selection of articles by two experienced researchers (Gough et al., 2013; Wilson, 2014).

The searches in ERIC and in the Fachinformationssystem Bildung for English- and German-language literature identified over 6000 studies, while the searches in Norart, Idunn, BibliotekDK and DIVA identified over 4000 studies.¹ After the process of reviewing the literature based on the criteria for inclusion and exclusion, the body of relevant English, German and Scandinavian studies was reduced to 129. Figure A1 illustrates the review process of the initial literature searches, which involved screening and the inclusion and exclusion of primary studies.

Figure A1. Search and screening process of the studies written in English, German and Scandinavian languages (searches in ERIC and in Fachinformationssystem Bildung, Idunn, Norart, Artikelsök, BibliotekDK, DIVA).

Appendix 4.

Categories and codes used in this study

Table

Characteristics
1	Source	A: ERIC
		B: Scandinavian database
		C: German database
		D: handsearch
		E: other
2	Publication type	A: peer-reviewed article
		B: book
		C: PhD thesis
3	Name of publication	Title
4	Title article	Title
5	Year	Year
6	Author(s)	Name(s)
7	First author’s country of origin	Affiliation country
8	Language	1: English
		2: Scandinavian
		2N: Norwegian
		2S: Swedish
		2D: Danish
		3: German
Context of the study
9	School level	A: primary
		B: secondary
		BL (lower)
		BU (upper)
		D: whole system
10	Actor focus	1: student
		2: teacher
		3: school leader
		4: local authority (when more than one, list the relevant actors: 1, 2, 3)
Concepts of the study
11	Research question, aim	Extract of text
Methods and quality
12	Type of study	A: theoretical/analytical
		B: empirical
		C: effect/intervention
13	Method	1: observation
		2: interview
		3: document analysis
		4: survey
		5: review
		6: case/quasi-experiment
		(when more than one, list the relevant methods: 1, 2, 3, 4, 5, 6)
14	Data and selection	Extract of text
15	Results	Extract of text