A Framework for Addressing Instrumentation Biases When Using Observation Systems as Outcome Measures in Instructional Interventions

Abstract

Many educational interventions seek to change teachers’ instructional practice. Standards-based observation systems are a common and useful tool to understand these interventions’ impact, but the process of measuring instructional change with observation systems is highly complex. This paper introduces a framework for examining and understanding potential instrumentation biases that arise when evaluations use observation systems to understand instructional change. The framework systematizes two processes that all studies must undertake: (1) the process of operationalizing the construct of teaching quality, and (2) the process of data collection. A study that engages in these processes generates observation scores that capture their own raters’ perspectives on specific segments of instruction. These scores must be generalized to draw conclusions about the intended constructs and settings. Systematizing these two processes highlights the necessary steps of a validity argument supporting evaluation conclusions and the instrumentation biases that threaten such conclusions. The framework is illustrated with an example from our recent work, which sought to understand instructional change since the adoption of the Common Core State Standards (CCSS).

Keywords:

• Classroom observation
• instructional change
• theoretical framework

Disclosure statement

No potential conflict of interest was reported by the author(s).

Author Note

All views expressed in this paper are those of the authors. A previous version of this work was presented at the AEFP 45th Annual Conference.

Open Research Statements

Study and Analysis Plan Registration

There is no registration associated with the case study reported in this manuscript.

Data, Code, and Materials Transparency

The data and code supporting analyses for the case study reported in this manuscript are not publicly accessible. Due to the nature of this research, participants did not agree for their data to be shared publicly.

Design and Analysis Reporting Guidelines

Not applicable.

Transparency Declaration

The lead author (the manuscript’s guarantor) affirms that the manuscript provides an honest, accurate, and transparent account of the case study analyses presented as an example of using the framework discussed in this manuscript.

Replication Statement

Not applicable.

Additional information

Funding

Work on this paper was supported by grants from the Spencer Foundation and the W.T. Grant foundation for a project entitled “Under Construction: The Rise, Spread, and Consequences of the Common Core State Standards Initiative in American Education.”