A comparison of methods for evaluating schools with respect to growth of students in subpopulations in three-level models

ABSTRACT

Although there is recognition that there may be differential outcomes for groups of students within schools, examination of outcomes for subgroups presents challenges to researchers and policymakers. It complicates analytic procedures, particularly when the number of students per school in the subgroup is small. We explored five alternatives for applying a three-level multilevel growth modeling framework to examine school-level achievement for a select subgroup of students (students with disabilities) using a large longitudinal database tracking reading achievement. The alternatives vary in terms of use of subgroup only or all student data, use of student-level predictors, and method of linking student or school-level outcomes to school effectiveness indices. Correlations from .57 to .99 among alternatives suggest the choice of how to derive school-level outcomes for a subgroup has consequences for inferences about the school’s effectiveness with the subgroup. Researchers’ assumptions and data available should guide the selection of an approach.

KEYWORDS:

• Multilevel growth modeling
• school’s effectiveness
• subgroups of students

Acknowledgements

The data set used in these analyses was created as part of a Cooperative Service Agreement from the Institute of Education Sciences (IES) to the University of Oregon establishing the National Center on Assessment and Accountability for Special Education (NCAASE; PR/Award Number R324C110004). The findings and conclusions do not necessarily represent the views or opinions of the U.S. Department of Education.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1 In addition, to investigate an alternative characterization of schools, we analyzed the ranks of the school-level estimates, and computed the root-mean-square differences of the ranks under each of the methods. The patterns, in terms of which methods produced relatively (dis)similar ranks, corresponded to the scatterplots in Figure 2.

Additional information

Funding

This work was supported by the National Center on Assessment and Accountability for Special Education under Grant number R324C110004.

Notes on contributors

Yixing Liu

Yixing Liu is an associate professor in the School of Management at Beijing University of Chinese Medicine. She is a quantitative methodologist who is primarily working on developing latent variable modeling approaches and applying them in studying children’s academic development.

Roy Levy

Roy Levy is a professor in the T. Denny Sanford School of Social and Family Dynamics, specializing in measurement and statistical analysis. His research and teaching interests include methodological investigations and applications in psychometrics and statistical modeling, focusing on item response theory, structural equation modeling, and Bayesian approaches to inference and modeling. He also works in areas of assessment design, focusing on evidentiary principles and applications in simulation-based assessments. For more on his work, visit https://sites.google.com/a/asu.edu/roylevy/

Nedim Yel

Nedim Yel is a lecturer in the Department of Counseling and School Psychology at University of Massachusetts Boston. His research focuses on quantitative methods as they apply to social, psychological, behavioral, and educational sciences. He specially focuses on hierarchical linear models, Bayesian modeling, and item response theory.

Ann C. Schulte

Ann C. Schulte is an Emeritus Professor in the Department of Psychology at North Carolina State University. Dr. Schulte’s research focuses on school-based services for children with disabilities and improvement of achievement outcomes for this population.