Evaluating the Impact of Rater Effects on Behavior Rating Scale Score Validity and Utility

Abstract

Behavior rating scales represent one of the most commonly used types of assessments in school psychology. Yet, they suffer from a fundamental limitation: They are an indirect methodology influenced partially by student behavior and partially by rater perspectives. Thus, the current study utilized advanced analytic approaches to evaluate rater effects on the Academic Competence Evaluation Scales–Short Form–Teacher (ACES-SF-T) with a partially crossed sample of 132 fourth- and fifth-grade students rated by seven teachers. Results indicated that rater effects had a minimal impact on the predictive validity of ACES-SF-T scores for state achievement tests, but at the individual level, rater effects could lead to starkly different conclusions about students’ academic, social, and behavioral functioning. Implications for research and practice are discussed.

Keywords:

• rater effects
• many-facet Rasch measurement
• socioemotional and behavioral assessment
• academic competence evaluation scales

DISCLOSURE

The authors have no conflicts of interest to report.

Notes

1 These examples assume consideration of bivariate relations, but the concepts extend to multivariate considerations as well.

2 For further reference on the differences in these ­approaches, see Engelhard & Wind, 2017; Linacre, 1996; Stahl 1994; Styck et al. (in press); and Sudweeks et al. 2004.

3 Most behavior rating scales use some type of polytomous ordinal scale to measure targeted student behaviors that contain 2 rating scale categories. Therefore, in addition to overall item difficulty (the location of the point at which there is a .50 probability of the highest versus the lowest rating scale category being assigned), the difficulty of item thresholds must also be modeled (the location of the point at which there is a .50 probability of a rating in category k versus k − 1).

4 As mentioned previously, to support MFRM, “fully crossed” data are not required (i.e., a data set in which every teacher rated every student); rather, the data set must have “connectivity,” which requires that every student and teacher be at least indirectly comparable. For our data, although it was possible to have every student rated by multiple teachers, such a data configuration would have been burdensome and was not necessary to establish connectivity. Thus, not every student was rated by every teacher, but sufficient numbers of cases were rated by multiple teachers to support connectivity. For a more detailed discussion of the data configurations needed to support MFRM, see Styck et al. (in press).

5 Although each student could have theoretically been rated by 3 teachers, such a design would have been time prohibitive and unnecessary to support MFRM (Linacre & Wright, 2002).

6 More extensive results and tables outlining specific details of our MFRM are available as supplemental materials.

Additional information

Notes on contributors

Christopher J. Anthony

Christopher J. Anthony, PhD, is an assistant professor in the School of Special Education, School Psychology, and Early Childhood Studies in the College of Education at the University of Florida. His research focuses broadly on improving the assessment of positive student competencies, especially academic enablers and social and emotional learning.

Kara M. Styck

Kara M. Styck, PhD, is an assistant professor in the Psychology Department in the College of Liberal Arts and Sciences at Northern Illinois University. Her research focuses on how ­psychological measurement impacts educational decisions for students (e.g., screening, progress monitoring, diagnostic decisions).

Erin Cooke

Erin Cooke, MEd, is the fourth- and fifth-grade learning community leader at P.K. Yonge Developmental Research School in Gainesville, Florida. She is also pursuing her EdD in curriculum, teaching, and teacher education in the College of Education at the University of Florida. Her passions in the field include teacher professional learning, mathematics and reading interventions, and social and emotional learning.

Justin R. Martel

Justin R. Martel, MS, is a doctoral student in the School Psychology Program at the University of Florida. His research interests include student athlete mental health, mindfulness-based therapy applications, and performance psychology.

Katherine E. Frye

Katherine E. Frye, MEd, is a doctoral student in the School Psychology Program at the University of Florida. Her research interests include social–emotional assessment and intervention as well as the promotion of noncognitive factors in K–12 settings.