Research on Data Use: A Framework and Analysis

Abstract

One of the central lessons from research on data use in schools and school districts is that assessments, student tests, and other forms of data are only as good as how they are used. But what influences how they are used? This relatively straightforward question turns out to be fairly complex to answer. Data use implicates a number of processes, conditions, and contexts. It involves interpretive processes, as using data requires that the user interpret the data and construct implications for next steps. It implicates social and organizational conditions, since the data use unfolds in the context of a multileveled organizational system. And, because data can be a source of power, particularly in the current accountability environment, data use also involves power relations. In this article, we put forward a framework for understanding the phenomenon of data use in the context of data use interventions. We draw on existing research and theory to identify key dimensions of data use that we should attend to and offer a way to understand how these dimensions might interact. In so doing, we provide guidance for studying the pathways between data use interventions and various outcomes of value.

Keywords:

• conceptual framework
• data use
• interpretation
• power
• social context
• social interaction

ACKNOWLEDGEMENTS

We are grateful to the Spencer Foundation for support for writing this article. We thank Andrea Conklin Bueschel, Paul Goren, Judith Warren Little, Pamela Moss, and 2 anonymous reviewers for helpful comments. An earlier draft of this article was developed to guide a series of convenings of scholars of data use sponsored by the Spencer Foundation under the auspices of their Data Use and Educational Improvement Initiative. We are grateful to the participants in the convenings for rich and challenging conversations that enriched our understanding of the facets of data use and greatly improved the article.

Notes

¹This article is meant to be a conceptual essay, not a comprehensive literature review. However, we did review a great deal of research on data use. We were aided in our efforts to identify relevant literature by the fact that the production of this essay happened in the context of a series of meetings sponsored by the Spencer Foundation that sought to characterize the existing research base on data use. As part of this initiative, the Spencer Foundation commissioned comprehensive literature reviews by 15 scholars on different aspects of the data use puzzle. We drew on these literature reviews (especially, Henig, in press; Honig & Venkateswaran, in press; Jennings, in press; Koretz & Jennings, 2010; Little, in press; Marsh, in press; Spillane, in press; and Supovitz, 2011), as well as a comprehensive literature review we had recently completed ourselves on data use at the district level (Coburn, Honig, & Stein, 2009) as a starting point. We read and reviewed articles identified by these manuscripts, as well as additional articles found in the reference lists of the articles we read. We also read and reviewed several recent edited volumes on data use (Bransford, Stipek, Vye, Gomez, & Lam, 2009; Earl & Timperley, 2009; Kowalki & Lasley, 2009; Mandinach & Honey, 2008; Moss, 2007) as well as a recent comprehensive review commissioned by the Institute for Education Sciences (Hamilton et al., 2009). These volumes were sources of scholarship on data use to review, and their reference lists provided insight into further resources to pursue. All told, we read and reviewed 161 articles, books, book chapters, and reports. However, while our review of the research was extensive, we do not claim that it was comprehensive. Indeed, it is possible that we missed some articles, especially given the burgeoning interest in this topic. Therefore, claims that we make about the state of the research literature on data use should be viewed with caution.

²In other work, we have referred to this construct as “the practice of data use” (Coburn & Turner, in press). Here, we refer to it as “the process of data use” to help distinguish individuals' engagement with data from the change in practice that may result.

³In focusing on data use interventions, this article focuses on planned or intentional change efforts. We acknowledge, however, that some data use may be unplanned or emergent.

⁴Although we do not discuss these here, other aspects of the organizational and political context that may influence data use include the following: formal positions and roles, such as data coaches or reading coaches (Lachat & Smith, 2005; Marsh et al., 2009); the hierarchical and differentiated organization of the school district central office and overall schooling system (Coburn et al., 2009; Thorn, Meyer, & Gamoran, 2007); rules and policies, such as specifications of how teachers use their non-instructional time in union contracts (Marsh et al., 2006); external organizations and actors (Burch & Hayes, 2009; Datnow & Honig, 2008); and trust (Herman & Gribbons, 2001; Ikemoto & Marsh, 2007; Ingram et al., 2004).

⁵In this section, we are focused on change in practice that results from engagement with data. However, it is important to note that data use is itself a form of practice in the Cook and Brown (1999) sense. Please see Coburn and Turner (in press) for a fuller treatment of the practice of data use, both conceptually and methodologically.

⁶Although, certainly, there could be more attention to the connection between comprehensive initiatives and student outcomes. A recent review of research on comprehensive data initiatives suggests that few studies attend to student outcomes and many that do have methodological problems that raise questions about their ability to draw conclusions about outcomes (Marsh, in press).