Quantifying Error in Survey Measures of School and Classroom Environments

REFERENCES

• Bill & Melinda Gates Foundation. (2010). Learning about teaching: Initial findings from the Measures of Effective Teaching project. Retrieved from http://www.metproject.org/downloads/Preliminary_Findings-Research_Paper.pdf
   Google Scholar
• Bliese, P. D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analyses. In K. J. Klein & S. W. J. Kozlowski ( Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions ( pp. 349–381). San Francisco, CA: Jossey-Bass.
   Google Scholar
• Bollen, K. A., & Lennox, R. (1991). Conventional wisdom on measurement: A structural equation perspective. Psychological Bulletin, 110, 305–314.
   Web of Science ®Google Scholar
• Brennan, R. L. (1995). The conventional wisdom about group mean scores. Journal of Educational Measurement, 32(4), 385–396.
   Web of Science ®Google Scholar
• Brennan, R. L. (2001a). Generalizability theory. New York, NY: Springer-Verlag.
   Google Scholar
• Brennan, R. L. (2001b). urGENOVA (Version 2.1) [Computer software and manual]. Iowa City, IA: University of Iowa. Retrieved from http://www.education.uiowa.edu/casma/
   Google Scholar
• Butrymowicz, S. A. (2012, May 13). Student surveys for children as young as 5 years old may help rate teachers. The Washington Post. Retrieved from http://www.washingtonpost.com/local/education/student-surveys-may-help-rate-teachers/2012/05/11/gIQAN78uMU_story.html
   Google Scholar
• Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234–246.
   Web of Science ®Google Scholar
• Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Belmont, CA: Wadsworth Group.
   Google Scholar
• Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334.
   Web of Science ®Google Scholar
• Cronbach, L. J. (2004). My current thoughts on coefficient alpha and successor procedures. Educational Psychological Measurement, 64, 391–418.
   Web of Science ®Google Scholar
• Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: John Wiley.
   Google Scholar
• Croon, M. A., & van Veldhoven, M. J. (2007). Predicting group-level outcome variables from variables measured at the individual level: A latent variable multilevel model. Psychological Methods, 12(1), 45–57.
   PubMed Web of Science ®Google Scholar
• De Boeck, P. (2008). Random item IRT models. Psychometrika, 73, 533–559.
   Web of Science ®Google Scholar
• Ferguson, R. (2010, October 14). Student perceptions of teaching effectiveness. Retrieved from http://www.gse.harvard.edu/ncte/news/Using_Student_Perceptions_Ferguson.pdf
   Google Scholar
• Holfve-Sabel, M., & Gustafsson, J. (2005). Attitudes towards school, teacher, and classmates at classroom and individual levels: An application of two-level confirmatory factor analysis. Scandinavian Journal of Educational Research, 49(2), 187–202.
   Google Scholar
• Hox, J. J. (2010). Multilevel analysis: Techniques and applications. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
   Google Scholar
• Ingersoll, R. (2001). Teacher turnover and teacher shortages: An organizational analysis. American Educational Research Journal, 38(3), 499–534.
   Web of Science ®Google Scholar
• Iversen, G. R. (1991). Contextual analysis (Vol. 81). Newbury Park, CA: Sage.
   Google Scholar
• James L. R. (1982). Aggregation bias in estimates of perceptual agreement. Journal of Applied Psychology, 67, 219–229.
   Web of Science ®Google Scholar
• Jeon, M. J., Lee, G., Hwang, J. W., & Kang, S. J. (2009). Estimating reliability of school-level scores using multilevel and generalizability theory models. Asia Pacific Education Review, 10(2), 149–158.
   Web of Science ®Google Scholar
• Kane, M. T., & Brennan, R. L. (1977). The generalizability of class means. Review of Educational Research, 47, 267–292.
   Web of Science ®Google Scholar
• Kozlowski, S. W. J., & Klein, K. J. (2000). A multilevel approach to theory and research in organizations: Contextual, temporal, and emergent processes. In K. J. Klein & S. W. J. Kozlowski ( Eds.), Multilevel theory, research, and methods in organizations ( pp. 3–90). San Francisco, CA: Jossey-Bass.
   Google Scholar
• Ladd, H. (2011). Teachers’ perceptions of their working conditions: How predictive of planned and actual teacher movement? Educational Evaluation and Policy Analysis, 33(2), 235–261.
   Web of Science ®Google Scholar
• Lee, V. E., & Loeb, S. (2000). School size in Chicago elementary schools: Effects on teachers’ attitudes and students’ achievement. American Educational Research Journal, 37(1), 3–31.
   Web of Science ®Google Scholar
• Liu, X. S., & Ramsey, J. (2008). Teachers’ job satisfaction: Analyses of the Teacher Follow-up Survey in the United States for 2000–2001. Teaching and Teacher Education, 24(5), 1173–1184.
   Web of Science ®Google Scholar
• Lüdtke, O., Marsh, H. W., Robitzsch, A., & Trautwein, U. (2011). A 2×2 taxonomy of multilevel latent contextual models: Accuracy and bias trade-offs in full and partial error-correction models. Psychological Methods, 16(4), 444–467.
   PubMed Web of Science ®Google Scholar
• Lüdtke, O., Marsh, H. W., Robitzsch, A., Trautwein, U., Asparouhov, T., & Muthén, B. (2008). The multilevel latent covariate model: A new, more reliable approach to group-level effects in contextual studies. Psychological Methods, 13, 203–229.
   PubMed Web of Science ®Google Scholar
• Marcoulides, G. A. (1996). Estimating variance components in generalizability theory: The covariance structure analysis approach. Structural Equation Modeling, 3, 290–299.
   Web of Science ®Google Scholar
• Marsh, H. W., Lüdtke, O., Robitzsch, A., Trautwein, U., Asparouhov, T., & Muthén, B. (2009). Doubly-latent models of school contextual effects: Integrating multilevel and structural equation approaches to control measurement and sampling error. Multivariate Behavioral Research, 44, 764–802.
   PubMed Web of Science ®Google Scholar
• Mihaly, K., McCaffrey, D. F., Staiger, D. O., & Lockwood, J. R. (2013). A composite estimator of effective teaching. Retrieved from http://metproject.org/downloadsMET_Composite_Estimator_of_Effective_Teaching_Research_Paper.pdf
   Google Scholar
• Miller, A. D., & Murdock, T. B. (2007). Modeling latent true scores to determine the utility of aggregate student perceptions as classroom indicators in HLM: The case of classroom goal structures. Contemporary Educational Psychology, 32(1), 83–104.
   Web of Science ®Google Scholar
• Moir, E. (2009). Validity and reliability of the North Carolina Teacher Working Conditions Survey. Santa Cruz: The University of California at Santa Cruz New Teacher Center. Retrieved from ncteachingconditions.org/research2008
   Google Scholar
• Muthén, B. O., & Muthén, L. K. (2010). Mplus (version 6.11) [computer software] Los Angeles, CA: Muthen & Muthen.
   Google Scholar
• “NYC School Surveys.” ( n.d.). Retrieved from http://schools.nyc.gov/Accountability/tools/survey/default.htm
   Google Scholar
• Nye, B., Konstantopoulos, S., & Hedges, L. V. (2004). How large are teacher effects? Educational Evaluation and Policy Analysis, 26(3), 237–257.
   Web of Science ®Google Scholar
• Pinheiro, J., Bates, D., Saikat, D., Sarkar, D., & the R Development Core Team (2012). nlme: Linear and Nonlinear Mixed Effects Models. R package version 3.1–104.
   Google Scholar
• Preacher, K. J., Zyphur, M. J., & Zhang, Z. (2010). A general multilevel SEM framework for assessing multilevel mediation. Psychological Methods, 15, 209–233.
   PubMed Web of Science ®Google Scholar
• Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models (2nd ed.). Newbury Park, CA: Sage.
   Google Scholar
• Raudenbush, S. W., Bryk, A. S., Cheong,Y. F., & Congdon, R. T. (2004). HLM (version 6.0) [computer software]. Skokie, IL: SSI, International.
   Google Scholar
• Raudenbush, S.W., Martinez, A., Bloom, H., Zhu, P., & Lin, F. (2010). Studying the reliability of group-level measures with implications for statistical power: A six-step paradigm, University of Chicago Working Paper.
   Google Scholar
• Raudenbush, S. W., Rowan, B., & Kang, S. J. (1991). A multilevel, multivariate model for studying school climate with estimation via the EM algorithm and application to U.S. high-school data. Journal of Educational Statistics, 16, 295–330.
   Google Scholar
• Raudenbush, S., & Sadoff, S. (2008). Statistical inference when classroom quality is measured with error. Journal of Research on Educational Effectiveness, 1, 138–154.
   Google Scholar
• Raykov, T., & Penev, S. (2009). Estimation of maximal reliability for multiple-component instruments in multilevel designs. British Journal of Mathematical and Statistical Psychology, 62, 129–142.
   PubMed Web of Science ®Google Scholar
• Rivkin, S. G., Hanushek, E., & Kain, J. F. (2005). Teachers, schools, and academic achievement. Econometrica, 73, 417–458.
   Web of Science ®Google Scholar
• Rowan, B., & Correnti, R. (2009). Studying reading instruction with teacher logs: Lessons from the Study of Instructional Improvement. Educational Researcher, 38, 120–131.
   Web of Science ®Google Scholar
• Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer (Vol. 1). Newbury Park, CA: Sage.
   Google Scholar
• Shavelson, R. J., Webb, N. M., & Rowley, G. (1989). Generalizability theory. American Psychologist, 44(6), 922–932.
   Google Scholar
• Shin, Y., & Raudenbush, S. W. (2010). A latent cluster-mean approach to the contextual effects model with missing data. Journal of Educational and Behavioral Statistics, 35, 26–53.
   Web of Science ®Google Scholar
• Shrout, P. E., & Fleiss, J. L. (1979) Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 2, 420–428.
   Web of Science ®Google Scholar
• Sirotnik, K. A. (1980). Psychometric implications of the unit-of-analysis problem (with examples from the measurement of organizational climates). Journal of Educational Measurement, 17, 245–282.
   Web of Science ®Google Scholar
• Webb, N. M., & Shavelson, R. J. (2005). Generalizability theory: Overview. In B. S. Everitt & D. C. Howell ( Eds.), Encyclopedia of statistics in behavioral science (Vol. 2, pp. 717–719). Chichester, UK: John Wiley & Sons.
   Google Scholar
• Wei, X., & Haertel, E. (2011). The effect of ignoring classroom-level variance in estimating the generalizability of school mean scores. Educational Measurement: Issues and Practice, 30(1), 13–22.
   Google Scholar
• Woodhouse, G., Yang, M., Goldstein, H., & Rasbash, J. (1996). Adjusting for measurement error in multilevel analysis. Journal of the Royal Statistical Society. Series A (Statistics in Society), 159(2), 201–212.
   Google Scholar
• Woodruff, D. (1993). Linear models for item scores: Reliability, covariance structure, and psychometric inference (Vol. 93, No. 4). American College Testing Program.
   Google Scholar
• Zyphur, M., Kaplan, S., & Christian, M. (2008). Assumptions of cross-level measurement and structural invariance in the analysis of multilevel data: Problems and solutions. Group Dynamics: Theory, Research, and Practice, 12, 127–140.
   Web of Science ®Google Scholar