The Generalizability of Systematic Direct Observations Across Time and Setting: A Preliminary Investigation of the Psychometrics of Behavioral Observation

Abstract

This study examined the generalizability of systematic direct observation across setting and time. Participants included 14 students from an intact inclusionary fifth grade classroom. On-task/off-task behavior was directly observed using momentary time-sampling recording, twice a day, for 10 school days. Using Generalizability (G) theory, results suggested that of the total amount of variance present, individual variation among participants accounted for 62% of the observed variance and unexplained error 24%. Although adequate levels of interobserver agreement were attained, resulting reliability coefficients of the observational procedures were in the low.60s. Results are discussed in terms of both behavioral and classic measurement theory and implications for practice.

Additional information

Notes on contributors

John M. Hintze

John M. Hintze, PhD, is an Associate Professor of School Psychology at the University of Massachusetts at Amherst. His research interests are in the areas of curriculum-based measurement, school-based assessment, data analysis, and program evaluation.

William J. Matthews

William J. Matthews, PhD, is a Professor of School Psychology at the University of Massachusetts-Amherst. His research interests include curriculum-based measurement, behavioral observation, and school-based consultation.