Locating data use in the microprocesses of district-level deliberations

ABSTRACT

While there is an abundance of data-use literature available, there is still a need to develop methodological approaches for studying naturally occurring data use in decision-making processes over time. The central contribution of this paper is a strategy to understand the use of data in long-term observations of educational leaders’ policy-making deliberations. Using longitudinal and observational data, we created ‘decision trajectories’ that traced microprocesses of deliberation around specific decisions over time. We employed frame theory to locate when and how data entered decision-making processes within these trajectories. Our approach addresses the use of data as they arise in the context of longitudinal observations – a method that provides insight into how data may be used to inform, frame, or justify educational decisions.

KEYWORDS:

• Data use
• decision making
• frame theory
• qualitative methods
• longitudinal

In context

Educators and policymakers today are expected to use data – metrics about students, teachers, or other aspects of education – to improve teaching and learning for students. To better support educators’ effective uses of data, researchers need to know more about how they use them in decision making. We offer a methodology for studying when and how data are used to inform decision making by educational leaders. First, we argue that observing how educational leaders collaborate and make decisions about a given issue over time is a valuable way to study data use. Second, we show how our method takes into account the ways leaders frame educational issues to better understand the role data play in how people reason together during deliberations. We posit that this approach holds promise for future studies of data use in educational decision making.

There has been growing interest by scholars and policy-makers internationally in understanding and promoting data use in education. Studies have uncovered ways that teachers (e.g. Means, Chen, DeBarger, & Padilla, 2011; Young, 2006) and school leaders (e.g. Kerr, Marsh, Ikemoto, Darilek, & Barney, 2006; Luo, 2008) use data and how data-use initiatives can increase data-use capacities in schools (e.g. Farley-Ripple & Buttram, 2014; Huguet, Marsh, & Farrell, 2014; Marsh & Farrell, 2015). A smaller number of studies have attended to how data are utilised to inform decisions at the central school district level (e.g. Coburn & Talbert, 2006), where administrators use data in decisions about district-wide policy making. Our research focuses on this level of data use: policy-making by educational leaders in a large urban school district in the USA. Despite the growth of research on data use over the last decade, there is a critical need to expand beyond the typical methods currently applied (Little, 2012). Existing studies tend to rely on short-term data collection and employ methods that are not strongly grounded in theory. In this paper, we argue for a methodology that is unique in a number of ways: it is longitudinal, observational, and theoretical.

Below, we begin by defining what we mean by data use. We then explain in more detail the need for a new approach to studying data use in long-term deliberations¹ at the systems level. Next, we describe methodological processes we developed to analyse data use in education leaders’ deliberations. Our processes address a number of methodological shortcomings identified in current approaches to studying data use in education. We explain several important facets of our methods, most centrally the development of longitudinal decision trajectories and the application of frame theory. While the purpose of this paper is methodological in nature, we draw on data from a recent study to illustrate what can be gained from this method. We posit that these methods hold promise for future longitudinal studies of naturally occurring data use.

The need for a new approach

Before we address how our methodology fits within the broader landscape of data-use research, we must first outline a central construct in this manuscript: data. We consider data to be ‘collections of descriptive indicators of social conditions, policy processes, and outcomes’ (Henig, 2012, p. 5). In school districts, data may include measures related to student achievement, school performance, teaching practice, administrative data, and programme implementation, among others. School district leaders tend to encounter data in their raw form (e.g. print-outs of individual student scores on state assessment), although sometimes they are arrayed in charts, graphs, or tables. They can be invoked orally, in text, or in presentation formats. This definition of data is central to our research, and while this paper focuses on district leaders’ use of data, the methods described below potentially extend to data use at the school or classroom levels as well.

Studies of the role of data in decision making must be rooted in an understanding of the nature of decision making. We know from extant research that decision making is a social process of interpretation, negotiation, and persuasion (Coburn, Toure & Yamashita, 2008; Coburn, Bae & Turner, 2008). In this process, data do not speak for themselves; rather, people must actively make meaning of them and construct implications for action (Coburn & Talbert, 2006; Farley-Ripple, 2012; Kennedy, 1982). How educators use data in decision making is influenced by individual and collective beliefs, which shape how people interpret data and their consequences for policy solutions (Cho & Wayman, 2014; Park, Daly, & Guerra, 2013), as well as by the dynamics of social interaction (Spillane, 2012). In addition, people with different degrees of status and authority have varied influence in negotiations, with consequences for strategic decisions (Goodwin, 1993; Coburn et al., 2008). Finally, decisions ‘accrete’ (Weiss, 1980) and are often accomplished by people working together throughout the course of many individual and collective actions, over time.

This conceptualisation of decision making poses a methodological challenge for researchers studying data use. Analysing the process of decision accretion, for example, requires an approach sensitive enough to detect the subtle accrual of choices that occurs in real-world deliberations over time. Existing research on data use falls short in several ways. First, few studies make use of longitudinal data in their investigations of data use in decision making. Most existing studies use intermittent data collection methods, gathering data at one or several points in time.² For instance, Means, Gallagher, and Padilla (2007) conducted a point-in-time survey with teachers regarding their use of student data systems. Other projects have involved one-time interviews or focus groups (e.g. Wayman, Cho, Jimerson, & Spikes, 2012); occasional, informal observations (e.g. Datnow, Park, & Wohlstetter, 2007); or analysis of a single meeting without equal attention to the broader arc of a decision (e.g. Horn, Kane, & Wilson, 2015). Yet, a ‘snapshot’ approach – the collection of data during one or several moments in time – does not paint a complete picture of decision making. Deliberations in which data may enter conversations occur over an extended period. Thus, an analysis of deliberations falls short when data are collected in bursts. Longitudinal studies are necessary to investigate the role that data play in ongoing deliberations over time (Little, 2012).

Second, data use studies often rely primarily on interview or survey methods (e.g. Armstrong & Anthes, 2001; Datnow et al., 2007; Jimerson & Wayman, 2015; Means, Padilla, & Gallagher, 2010; Petrides & Nodine, 2005). While useful, interviews and surveys do not capture the nuance of data use in practice over time. Both rely on self-report, which can fall prey to social desirability bias and retrospective smoothing. When using surveys and interviews, it is also difficult to untangle issues of status, authority, power, and politics, all of which come into play when district leaders engage with one another while making decisions (Coburn et al., 2008; Johnson, 1999).

Third, conventional methods used to study data use are typically unable to identify ways in which data are leveraged within the microprocesses of decision making. Interviews and surveys are not able to capture the more nuanced back-and-forth that often occurs between actors throughout deliberations. Given that decision making is an interactive process, unfolding over time, across many settings, and with different configurations of actors in both formal and informal settings (Kennedy & Kennedy, 1982; Weiss, 1980; Weiss & Bucuvalas, 1980), observational methods are better suited to capture data use in decision making.

Finally, there is little research on data use that employs theories that are supported by what we know about the nature of decision making. Extant studies rarely focus on data as a means of persuasion within deliberations. However, the process of decision making is often one of argumentation, in which individuals are negotiating on behalf of a certain course of action (Asen, Gurke, Solomon, Conners, & Gumm, 2011; Spillane & Miele, 2007). For instance, using sustained observational methods, Coburn, Touré, and Yamashita (2009) argued that decision making in complex organisations like school districts is centrally about interpretation, argumentation, and persuasion. They noted that in these circumstances data were leveraged to paint a problem in a particular light or to provide evidence for a preferred solution.

These limitations led Little (2012) to call for ‘a more conceptually robust, methodologically sophisticated, and extensive programme of micro-process research on data use’ (p. 145). The field has increasingly been turning toward observational methods in data-use studies (e.g. Earl, 2008; Timperley, 2008). However, more work is needed to develop robust methods for designing for and analysing observational data. Here, we present a novel methodological approach that aims to address the current limitations in studies of data use in decision making.

Our methodological approach

Our approach to investigating data use in district-level decision making utilises longitudinal observational data, interviews, and document analysis. We developed these approaches in the context of a study of a mid-sized urban district in the USA. Educational leaders were engaged in decision-making efforts related to district-wide implementation of a new set of academic standards called the Common Core State Standards Mathematics (CCSS-M).³ Our research question was: when and how do data enter district leaders’ decision making in response to the CCSS-M? Collected data include observation field notes and video recordings of district-level planning meetings, transcripts from interviews with district leaders, and artefacts. These records allowed us to investigate when and how data entered discussions of CCSS-M implementation, the dynamics of interpretation and negotiation about the meaning and implications of data, and their consequences for key decisions. Two years of detailed field notes comprise the bulk of our observational dataset.⁴

To illustrate our approach, we share examples from the development of a particular policy in our study. Education leaders in one area of the district, called the Improvement Zone,⁵ needed to decide whether to keep a specific curricular programme, which we call ‘Math Basics’. The Improvement Zone was known for its history of low performance on standardised assessments. In the years prior to our study, the Improvement Zone had contracted with a traditional, back-to-basics partner for mathematics instructional materials, coaching, and training. This programme, Math Basics, had resulted in some notable gains in students’ academic achievement, but it was a curriculum that had scripted lessons for teachers. This approach was not congruent with the direction that district leaders felt their curriculum should be moving: toward more critical thinking for both students and teachers. In addition, this existing programme appeared to be incongruent with the new CCSS-M. Therefore, district leadership was faced with a decision: continue to support the curriculum, modify it, or bring in a new approach. With any of these choices, they would also need to decide how to structure teacher support in mathematics for Improvement Zone schools in the upcoming year to improve mathematics teaching and learning.

As one district leader, Miguel, pointed out in an early meeting, the fact that some Improvement Zone schools using Math Basics had seen substantial growth on state assessments complicated this decision. Several leaders brought a commitment to helping teachers use instructional approaches that reflected the new conceptually focused standards. The overarching questions became whether to keep Math Basics, what the alternative options might be, and how they might adapt and improve the programme to better fit the new academic standards. In the following sections, we use the Math Basics deliberation to present the methodological approach we developed.

Studying decision making over time: leveraging ‘decision trajectories’

Analysis of decision making in situ presents a methodological challenge, as extensive longitudinal data require a systematic approach for narrowing information to only that which is relevant. To manage this challenge, we created collections of data around what we titled ‘decision trajectories’, and segmented observation data into smaller units of analysis called ‘episodes’. In this section, we explain the theory behind and utility of segmenting longitudinal data into trajectories and then episodes, and provide examples of the process from the Math Basics trajectory.

Decision trajectories

The decision trajectory is a way to conceptualise the theoretical understanding of decision making as a nonlinear process that takes place over time. We identified distinct policy issues that district leaders deliberated about over the course of the dataset and coded all the observation field notes which included discussion of these key issues. In so doing, our team created sets of observation field notes about specific decisions from meetings that occurred across a two-year span, constructing datasets that represented ‘decision trajectories’. The Math Basics policy decision was one of these trajectories and included two years of field notes, which captured leaders’ deliberation around the math programme decision.

There were several advantages to arranging observational data into chronological decision trajectories. Doing so organised data around a set of topics that related to a consequential policy deliberation over time, giving us the ability to identify shared understandings around a single topic that were built among district personnel over time. Each decision trajectory provided a view of developments and changes of perspectives related to, for example, the benefits and drawbacks of using the Math Basics programme.

Further, because of our familiarity with Improvement Zone deliberations over time using our decision trajectory, we could identify not only references to data that were explicit – such as a discussion of state assessment scores – but also references that were less obvious. Invocations of data may appear more implicit in some cases because of district leaders’ shared knowledge. For instance, when a district leader mentioned in a meeting that ‘Math Basics got results in Steinbeck [School],’ we were able to tie this reference back to earlier data conversations about how Math Basics improved state assessment scores in mathematics for low-income students of colour at the mentioned school.

Decision trajectories were a theoretically and practically useful strategy to organise field notes chronologically by policy topic. Within these large collections that we called decision trajectories, it was necessary to define smaller units of analysis that could be used to analyse how district leaders leveraged data within their deliberations.

Episodes

Within each trajectory, we identified smaller divisions that we called episodes, which were segments of field notes containing sustained discussions around a given topic. We considered a segment ‘sustained’ if it included at least five contributions to the conversation on the same topic. Typically, there were multiple episodes within a given meeting, as people tended to shift topics in planned (i.e. per the agenda) or emergent ways.

We then analysed the micro-processes of deliberation at the episode level. In one episode in the Math Basics trajectory, a meeting participant initiated a conversation about Math Basics, reporting to the group that ‘I’m hearing from principals and assistant principals that people are worried about using Math Basics this year.’ A discussion followed as people contributed accounts of the problem and possible solutions. Analysing deliberation within episodes made it possible to focus more closely on these moments and connect them to past and future discussions of the same issues.

Analysing the content and flow of deliberation with frame theory

To analyse the finer-grained details within episodes, we developed an analytic approach rooted in frame theory. Social movement scholars use frame theory to explain how ideas are produced and invoked to mobilise people to action (Goffman, 1974; McAdam & Scott, 2005; Snow & Benford, 1988; Snow, Rochford, Worden, & Benford, 1986). Framing is used to characterise a problem in a particular way, persuade others to one’s opinion, or to counter others’ claims. Scholars have previously used frame theory and similar approaches to investigate school board deliberations (Asen et al., 2011; Tracy & Dimock, 2004), local negotiations of policy implementation (Coburn, 2006; Coburn & Woulfin, 2012), and – most relevant to our work – data use in decision making (Coburn et al., 2009; Park et al., 2013).

Frame theory and the role of data

Frame theory illuminates how ideas are produced and invoked to mobilise people to action (Snow & Benford, 1988; Snow et al., 1986). Framing involves noticing, punctuating, and organising information into an explanation that renders complexity meaningful. It is an interactive process. Frame theory helps us understand the efforts of actors in social settings to persuade others to view problems in a certain light (diagnostic or problem frames) and to pursue particular courses of action to address those problems (prognostic or solution frames). Frames can be contested or countered as others put forth alternative portrayals of the situation or paths to pursue (Stone, 1988). Individuals and groups attempt to construct ways of framing a problem that create resonance, or a ‘deep responsive chord’ (Binder, 2002, p. 220), that motivates others to join in support of an interpretation or solution.

Data and other forms of information can play a central role in frame dynamics. They act as the ‘raw material used by policy actors to frame, support, oppose, and justify policy arguments in various decision-making arenas’ (Johnson, 1999, p. 24). That is, data are central to what scholars call the ‘reason giving’ at the core of deliberation in policy contexts (Asen & Gurke, 2014). When one presents a frame in deliberation, it is often accompanied by a reason that justifies or otherwise bolsters its relevance or importance. Given our focus on data use, we paid particular attention to the role of data and other forms of information in the reasons that people provided.

Identifying frames and reasons

In analysing episodes of deliberation, we focused on frames and reasons that education leaders put forth as they argued for a specific problem definition or solution – that is, how they marshalled support for their arguments. In particular, we attended to when and how individuals invoked data in these deliberations, and with what consequences. To do so, we noted each instance of framing and reason giving in each episode within the decision trajectory.

By analysing participant conversations frame-by-frame, we saw how and when speakers brought up data in deliberations to add weight to a particular problem definition or as a ‘call to action’ solution frame. For example, one leader, Miguel, emphasised that there was a problem with Math Basics and defined that problem through the use of data: ‘[Math Basics curriculum] initially brought math scores up, but then they plateaued, especially for English Learners.’ Here, state achievement data were used by Miguel as part of the reasoning to support his frame of the problem with the Math Basics programme: while there may have been initial growth in student performance, the programme did not support long-term learning, especially for certain groups of students.

Mapping frames to understand invocations of data

To better understand deliberations identified by frame analysis, we developed a visual representation to map relevant frames and reasons within each episode. For each episode, we first identified all problem and solution frames and their reasons. Then, we mapped the frames chronologically, from left to right across the page. We transferred episode text to these maps (see Figure 1), which provided a visual representation of the conversations taking place in key meetings, highlighting the back-and-forth nature of decision making. The maps thus distilled the content of argumentation within deliberations. By mapping these microprocesses, we isolated instances where data entered deliberations as located within frames and reasons.

Identifying where data entered the conversation using maps clarified the role data played in framing. We were able to investigate questions such as: to what degree were data used in reasons that justified solutions frames compared to problem frames? Were data more likely to be invoked in concert with other reasons, or independently? Was there a pattern in the conversation following the invocation of data? Were data invoked more by some people, or people in certain roles, than others? These important questions were possible to explore using frame theory, aided by our mapping.

In sum, while the over-arching decision trajectories provided context, it was at the episode level that we mapped when, where, and how problem and solution frames were put forth and data were invoked in deliberations. In these ways, we applied frame theory to longitudinal observations of naturally occurring data use by district leaders as they made policy decisions.⁶

Benefits of leveraging the methodology: the Math Basics deliberation

To illustrate the application of this methodology, we describe a set of key frames and reasons that arose in the Math Basics deliberation. In a meeting early in the trajectory, a small group of Improvement Zone leaders met with a long-time research partner to talk about how to support math instruction in the Improvement Zone. At one point the outside partner, Quentin, asked Ellen, an Improvement Zone administrator, ‘Are we going to keep paying for Math Basics next year?’ It appeared that the decision was primarily Ellen’s to make as leader of the Improvement Zone, and she had not yet made it. In response, Ellen offered three reasons to Quentin, two of which seemed to suggest not continuing while one suggested continuing. She first anticipated future consequences, saying, ‘The original contract with Math Basics won’t be there next year.’ Secondly, she responded on behalf of stakeholders in the Improvement Zone region, ‘One school doesn’t want it.’ Lastly, Ellen here invoked data in stating, ‘Math scores have gone up in schools using Math Basics.’ Because we were observing this meeting, we also noted that Ellen and Quentin looked through a binder that showed state test scores for schools that had and had not used the Math Basics approach. Further, there was an additional frame in support of keeping the programme; in defence of keeping Math Basics, Ellen stated, ‘We have a lot of mediocre teaching’ in the Improvement Zone, and the current programme was easy for teachers to implement. From later meetings, we know that both Quentin and Ellen were concerned with the idea of removing curricular supports from teachers without an alternative math programme in place that reflected the upcoming academic standards. Figure 2 illustrates what this interaction looked like when we isolated the contributions through mapping.

Figure 1. An excerpt from the map of an observed deliberation. In our mapping process, problem frames were represented by squares, while solutions and reasons in support of solutions were denoted by circles. Frames that were in support of a previous frame were connected with solid lines, while counterframes were connected with dashed lines.

Figure 2. An excerpt of a central episode in the Math Basics trajectory.

Quentin and Ellen concluded, looking at the data binder, that the programme had raised scores in schools. These data supported one of Quentin’s solution frames that ‘you need to build on Math Basics, not repudiate it.’ So, even though Ellen saw Math Basics as overly scripted and skills-focused – not compatible with the new academic standards or her beliefs about good instructional approaches – she and other Improvement Zone leaders chose to continue supporting Math Basics. In later meetings, Ellen and others critiqued aspects of the programme, but they continued to frame their work as ‘modifying’ and ‘building on’ it, and this solution frame informed their ongoing planning of professional development about math instruction for teachers and principals in the Improvement Zone. By analysing the decision, which occurred over time and across multiple meetings, we found that the data in the early meeting, which showed that schools using Math Basics had increases in standardised test scores, held significant weight in the deliberation.

This example helps to illustrate the benefits of our methodological approach. Analysing frame dynamics allowed us to track how problem and solution frames related to Math Basics were reiterated or changed over time. Had we only observed and analysed a single meeting, we may have concluded that Math Basics would be rejected in favour of a more fitting curriculum. Or, that data played a limited role because they were not explicitly referred to in a given meeting. Looking at frames across meetings, we saw how the dynamics shifted, as the team decided to keep the programme for the upcoming school year, and how that decision later opened new problems to solve – including how to adapt the programme, how to fit it with other instructional materials being used, and how to design professional development for teachers and school leaders.

Contributions of the approach

The approach outlined here makes several methodological contributions. Our methods leverage longitudinal and observational data in analysing naturally occurring data use in the district context. We provide guidelines for distilling the broad datasets, beginning with the creation of decision trajectories, then moving to the episode as the analytic unit. Since each of these constructs is rooted in understandings of the nature of decision making and persuasion, this approach can provide a model for using observation methods in studying data use in a conceptually grounded way. The construction of decision trajectories from longitudinal, observational data offered important advantages over other approaches. By grouping all observational data related to a given decision and then further segmenting into episodes, we not only were able to isolate topics of interest from the vast stream of district policymaking, but we could also track them over time and across meeting spaces. Utilizing the Math Basics trajectory, for instance, allowed us to analyse all talk related to Math Basics without wading through the hundreds of hours of observations about other issues occurring in the district.

Trajectories also allowed us to see ways that deliberations about particular issues unfolded over time, rather than treating decisions as discrete events. In district-leader meetings, participants talked about a variety of topics, often in a nonlinear way, with some topics being connected to and interdependent with other topics. Creating a decision trajectory out of episodes of topical deliberation helped us see how education leaders made sense of specific issues over time and decided on next steps in ways that shaped future work. In the case of Math Basics, for example, looking beyond a single observation or meeting made it possible to understand how invocations of data fit in the larger context.

Our methodological contributions build on previous work by Coburn (2006), as we have further developed an approach to analytically leverage frame theory to isolate and analyse the role that data play in policy deliberations. By concurrently analysing for frames and reasons, we contribute to research on data use by joining and operationalizing the concepts of decision trajectories and frame theory in deliberation. A focus on framing within decision trajectories allows us to ask and answer questions such as: how do particular problem framings get invoked over time in different phases of policy deliberations? When and how are particular solution frames put forward, and which ones carry through into policy-making, with what adjustments over time? What are the role of data in understanding the problems and solutions educators are deliberating?

In sum, this paper provides a methodological account of our approach to studying uses of data in deliberations over time. Unlike prior research, our analyses focus on naturally occurring uses of data as they arise in the context of longitudinal observations – an approach that has great potential to provide insights into how data may inform educational decisions at policy-making and other educational levels. This methodological approach offers scholars new insights into the role of data in framing and justifying responses to important policies at the district level.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This work was supported by the Spencer Foundation [201500139].

Notes

1. We use the term deliberation to refer to discussions – centred on problems or choices to be made – that occurred within and often extended across district meetings.

2. There are exceptions, including Cosner (2011), and our own prior work (e.g. Coburn et al., 2008; Coburn et al., 2009).

3. Our dataset was gathered from August 2012 through September 2014 when we observed district meetings related to mathematics over two years across a range of district subunits (n = 300 h). We interviewed all mathematics leaders and all other district administrators who had instructional responsibilities in mathematics (n = 87 interviews with 44 administrators). We gathered 1826 artefacts, including documents from the meetings we observed and those related to district mathematics policy. The comprehensiveness of data collected contributes significantly to the methods described herein, as their longitudinal and observational nature supported the novel analytic techniques we explain.

4. We recognise that not all researchers are able to invest the resources necessary to conduct the time-intensive data collection we recommend here.

5. All names – of districts, people, and programmes – are pseudonyms.

6. During each step of the coding process, we developed criteria for our analytical decisions, including what constitute an episode, a frame, and a reason. These were explicit criteria that our team used as guides with which to frequently calibrate our coding – a process of coding in groups, individually, and then regrouping to check one another’s work. This is an approach we suggest for achieving reliability using this methodology.