School self-evaluation instruments and cognitive validity. Do items capture what they intend to?

References

• Alwin, D. F. (1991). Research on survey quality. Sociological Methods & Research, 20, 3–29. doi:10.1177/0049124191020001001
   Web of Science ®Google Scholar
• Alwin, D. F. (2010). How good is survey measurement? Assessing the reliability and validity of survey measures. In P. V. Marsden & J. D. Wright (Eds.), Handbook of survey research (2nd ed., pp. 405–434). Bingley, UK: Emerald Group.
   Google Scholar
• Antoniou, P., Myburgh-Louw, J., & Gronn, P. (2016). School self-evaluation for school improvement: Examining the measuring properties of the LEAD surveys. Australian Journal of Education, 60, 191–210. doi:10.1177/0004944116667310
   Web of Science ®Google Scholar
• Bass, B. M., & Riggio, R. E. (2006). Transformational leadership (2nd ed.). Mahway, NJ: Lawrence Erlbaum Associates.
   Google Scholar
• Bateson, N. (1984). Data construction in social surveys. London, UK: George Allen & Unwin.
   Google Scholar
• Beatty, P. C., & Willis, G. B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly, 71, 287–311. doi:10.1093/poq/nfm006
   Web of Science ®Google Scholar
• Belson, W. A. (1981). The design and understanding of survey questions. Aldershot, UK: Gower.
   Google Scholar
• Bliese, P. D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extentions, and new directions (pp. 349–381). San Francisco, CA: Jossey-Bass.
   Google Scholar
• Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234–246. doi:10.1037/0021-9010.83.2.234
   Web of Science ®Google Scholar
• Chen, G., Mathieu, J. E., & Bliese, P. D. (2004). A framework for conducting multi-level construct validation. In F. J. Yammarino & F. Dansereau (Eds.), Multi-level issues in organizational behavior and processes (Vol. 3, pp. 273–303). Oxford, UK: Elsevier.
   Google Scholar
• Christensen, R. H. B. (2015). Ordinal – Regression models for ordinal data (R package Version 2015.6-28). Retrieved from https://cran.r-project.org/web/packages/ordinal/ordinal.pdf
   Google Scholar
• Cohen, L., Manion, L., & Morrison, K. (2011). Research methods in education (7th ed.). London, UK: Routledge.
   Google Scholar
• Collins, D. (2003). Pretesting survey instruments: An overview of cognitive methods. Quality of Life Research, 12, 229–238. doi:10.1023/a:1023254226592
   PubMed Web of Science ®Google Scholar
• Conrad, F., Blair, J., & Tracy, E. (1999, November). Verbal reports are data! A theoretical approach to cognitive interviews. Paper presented at the Federal Committee on Statistical Methodology Research Conference, Arlington, VA.
   Google Scholar
• DeMaio, T. J., & Landreth, A. (2004). Do different cognitive interview techniques produce different results? In S. Presser, J. M. Rothgeb, M. P. Couper, J. T. Lessler, E. Martin, J. Martin, & E. Singer (Eds.), Methods for testing and evaluating survey questionnaires (pp. 89–108). Hoboken, NJ: John Wiley & Sons.
   Google Scholar
• Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (Rev ed.). Cambridge, MA: The MIT press.
   Google Scholar
• Fleiss, J. L. (1981). The measurement of interrater agreement. In Statistical methods for rates and proportions (2nd ed., pp. 212–236). New York, NY: John Wiley & Sons.
   Google Scholar
• Fowler, F. J., Jr. (1992). How unclear terms affect survey data. Public Opinion Quarterly, 56, 218–231. doi:10.1086/269312
   PubMed Web of Science ®Google Scholar
• Galesic, M., Tourangeau, R., Couper, M. P., & Conrad, F. G. (2008). Eye-tracking data: New insights on response order effects and other cognitive shortcuts in survey responding. Public Opinion Quarterly, 72, 892–913. doi:10.1093/poq/nfn059
   PubMed Web of Science ®Google Scholar
• Gisev, N., Bell, J. S., & Chen, T. F. (2013). Interrater agreement and interrater reliability: Key concepts, approaches, and applications. Research in Social and Administrative Pharmacy, 9, 330–338. doi:10.1016/j.sapharm.2012.04.004
   PubMed Web of Science ®Google Scholar
• Groves, R. M., Fowler, F. J., Jr., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2009). Survey methodology (2nd ed.). Hoboken, NJ: John Wiley & Sons.
   Google Scholar
• Hendriks, M. A. (2000). Kwaliteitszorg voortgezet onderwijs: Instrumenten en organisaties [Quality care in secondary education: Instruments and organizations]. Utrecht, The Netherlands: VVO/Q5, project kwaliteitszorg voortgezet onderwijs.
   Google Scholar
• Hendriks, M. A., & Bosker, R. (2003). ZEBO instrument voor zelfevaluatie in het basisonderwijs. Handleiding bij een geautomatiseerd hulpmiddel voor kwaliteitszorg in basischolen [ZEBO instrumentation for self-evaluation in primary education. Manual to the computerized instrumentation for quality care in primary education]. Enschede, The Netherlands: Twente University Press.
   Google Scholar
• Hinkin, T. R. (1995). A review of scale development practices in the study of organizations. Journal of Management, 21, 967–988. doi:10.1177/014920639502100509
   Web of Science ®Google Scholar
• Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17–64). Westport, CT: American Council on Education/Praeger.
   Google Scholar
• Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50, 1–73. doi:10.1111/jedm.12000
   Web of Science ®Google Scholar
• Karabenick, S. A., Woolley, M. E., Friedel, J. M., Ammon, B. V., Blazevski, J., Bonney, C. R., … Kelly, K. L. (2007). Cognitive processing of self-report items in educational research: Do they think what we mean? Educational Psychologist, 42, 139–151. doi:10.1080/00461520701416231
   Web of Science ®Google Scholar
• Klein, K. J., Conn, A. B., Smith, D. B., & Sorra, J. S. (2001). Is everyone in agreement? An exploration of within-group agreement in employee perceptions of the work environment. Journal of Applied Psychology, 86, 3–16. doi:10.1037/0021-9010.86.1.3
   PubMed Web of Science ®Google Scholar
• Koskey, K. L. K., Karabenick, S. A., Woolley, M. E., Bonney, C. R., & Dever, B. V. (2010). Cognitive validity of students’ self-reports of classroom mastery goal structure: What students are thinking and why it matters. Contemporary Educational Psychology, 35, 254–263. doi:10.1016/j.cedpsych.2010.05.004
   Web of Science ®Google Scholar
• Kozlowski, S. W. J., & Klein, K. J. (2000). A multilevel approach to theory and research in organizations: Contextual, temporal, and emergent processes. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 3–90). San Francisco, CA: Jossey-Bass.
   Google Scholar
• Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213–236. doi:10.1002/acp.2350050305
   Web of Science ®Google Scholar
• Krosnick, J. A., Narayan, S., & Smith, W. R. (1996). Satisficing in surveys: Initial evidence. New Directions for Evaluation, 1996(70), 29–44. doi:10.1002/ev.1033
   Google Scholar
• Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174. doi:10.2307/2529310
   PubMed Web of Science ®Google Scholar
• Lenzner, T. (2012). Effects of survey question comprehensibility on response quality. Field Methods, 24, 409–428. doi:10.1177/1525822x12448166
   Web of Science ®Google Scholar
• Lenzner, T., Kaczmirek, L., & Lenzner, A. (2010). Cognitive burden of survey questions and response times: A psycholinguistic experiment. Applied Cognitive Psychology, 24, 1003–1020. doi:10.1002/acp.1602
   Web of Science ®Google Scholar
• Lissitz, R. W., & Samuelsen, K. (2007). A suggested change in terminology and emphasis regarding validity and education. Educational Researcher, 36, 437–448. doi:10.3102/0013189x07311286
   Google Scholar
• MacBeath, J. (1999). Schools must speak for themselves: The case for school self-evaluation. London, UK: Routledge.
   Google Scholar
• MacBeath, J., Schratz, M., Meuret, D., & Jakobsen, L. (2000). Self-evaluation in European schools: A story of change. London, UK: RoutledgeFalmer.
   Google Scholar
• Maslowski, R. (2001). School culture and school performance: An explorative study into the organizational culture of secondary schools and their effects. Enschede, The Netherlands: Twente University Press.
   Google Scholar
• Mathieu, J. E., & Chen, G. (2011). The etiology of the multilevel paradigm in management research. Journal of Management, 37, 610–641. doi:10.1177/0149206310364663
   Web of Science ®Google Scholar
• McGee, G. W., & Ford, R. C. (1987). Two (or more?) Dimensions of organizational commitment: Reexamination of the affective and continuance commitment scales. Journal of Applied Psychology, 72, 638–641. doi:10.1037/0021-9010.72.4.638
   Web of Science ®Google Scholar
• McNamara, G., O’Hara, J., Lisi, P. L., & Davidsdottir, S. (2011). Operationalising self-evaluation in schools: Experiences from Ireland and Iceland. Irish Educational Studies, 30, 63–82. doi:10.1080/03323315.2011.535977
   Web of Science ®Google Scholar
• Meuret, D., & Morlaix, S. (2003). Conditions of success of a school’s self-evaluation: Some lessons of an European experience. School Effectiveness and School Improvement, 14, 53–71. doi:10.1076/sesi.14.1.53.13867
   Web of Science ®Google Scholar
• O’Muircheartaigh, C. (1999). CASM: Successes, failures, and potential. In M. G. Sirken, D. J. Herrmann, S. Schechter, N. Schwarz, J. M. Tanur, & R. Tourangeau (Eds.), Cognition and survey research (pp. 39–63). New York, NY: Wiley & Sons.
   Google Scholar
• Royston, P. N. (1989). Using intensive interviews to evaluate questions. In F. J. Fowler, Jr. (Ed.), Health survey research methods (pp. 3–7). Washington, DC: National Center for Health Services Research.
   Google Scholar
• Ryan, K., Gannon-Slater, N., & Culbertson, M. J. (2012). Improving survey methods with cognitive interviews in small- and medium-scale evaluations. American Journal of Evaluation, 33, 414–430. doi:10.1177/1098214012441499
   Web of Science ®Google Scholar
• Scheerens, J. (2008). Review and meta-analyses of school and teaching effectiveness. Berlin, Germany: Bundesministerium für Bildung und Forsching (BMBF).
   Google Scholar
• Schildkamp, K., Lai, M. K., & Earl, L. M. (Eds.). (2013). Data-based decision making in education: Challenges and opportunities. Dordrecht, The Netherlands: Springer.
   Google Scholar
• Schildkamp, K., Visscher, A., & Luyten, H. (2009). The effects of the use of a school self-evaluation instrument. School Effectiveness and School Improvement, 20, 69–88. doi:10.1080/09243450802605506
   Web of Science ®Google Scholar
• Schwarz, N. (2007). Cognitive aspects of survey methodology. Applied Cognitive Psychology, 21, 277–287. doi:10.1002/acp.1340
   Web of Science ®Google Scholar
• Tourangeau, R., & Bradburn, N. M. (2010). The psychology of survey response. In P. V. Marsden & J. D. Wright (Eds.), Handbook of survey research (2nd ed., pp. 315–346). Bingley, UK: Emerald Group.
   Google Scholar
• Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge, UK: Cambridge University Press.
   Google Scholar
• Vanhoof, J., Deneire, A., & Van Petegem, P. (2011). Waar zit beleidsvoerend vermogen in (ver)scholen? Aanknopingspunten voor zelfevaluatie en ontwikkeling [Where is policymaking capacity hidden in schools? Cruxes for self-evaluation and development]. Mechelen, Belgium: Plantyn.
   Google Scholar
• Vanhoof, J., & Van Petegem, P. (2010). Evaluating the quality of self-evaluations: The (mis)match between internal and external meta-evaluation. Studies in Educational Evaluation, 36, 20–26. doi:10.1016/j.stueduc.2010.10.001
   Google Scholar
• Van Petegem, P., Cautreels, P., & Deneire, A. (2003). IZES Basisonderwijs: Instrument voor zelfevaluatie van basisscholen [IZES primary education: Instrument for self-evaluation in primary schools]. Leuven, Belgium: Acco.
   Google Scholar
• Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. London, UK: Sage.
   Google Scholar
• Woolley, M. E., Bowen, G. L., & Bowen, N. K. (2006). The development and evaluation of procedures to assess child self-report item validity. Educational and Psychological Measurement, 66, 687–700. doi:10.1177/0013164405282467
   PubMed Web of Science ®Google Scholar