A multilevel study of position effects in PISA achievement tests: student- and school-level predictors in the German tracked school system

References

• Adams, R. J. (2005). Reliability as a measurement design effect. Studies in Educational Evaluation, 31, 162–172. doi:10.1016/j.stueduc.2005.05.008
   Google Scholar
• Asparouhov, T., & Muthén, B. (2012). Saddle points. Retrieved from http://www.statmodel.com/download/SaddlePoints2.pdf
   Google Scholar
• Asseburg, R., & Frey, A. (2013). Too hard, too easy, or just right? The relationship between effort or boredom and ability-difficulty fit. Psychological Test and Assessment Modeling, 55, 92–104.
   Google Scholar
• Baumert, J., Stanat, P., & Watermann, R. (2006). Schulstruktur und die Entstehung differenzieller Lern- und Entwicklungsmilieus [School structure and the development of differential environments for learning and development]. In J. Baumert, P. Stanat, & R. Watermann (Eds.), Herkunftsbedingte Disparitäten im Bildungswesen: Differenzielle Bildungsprozesse und Probleme der Verteilungsgerechtigkeit (pp. 95–188). Wiesbaden: VS für Sozialwissenschaften.10.1007/978-3-531-90082-7
   Google Scholar
• Brennan, R. L. (1992). The context of context effects. Applied Measurement in Education, 5, 225–264. doi:10.1207/s15324818ame0503_4
   Google Scholar
• Bulut, O., Lei, M., & Guo, Q. (2016). Item and testlet position effects in computer-based alternate assessments for students with disabilities. International Journal of Research & Method in Education. Advance online publication. doi:10.1080/1743727X.2016.1262341
   Google Scholar
• Debeer, D., Buchholz, J., Hartig, J., & Janssen, R. (2014). Student, school, and country differences in sustained test-taking effort in the 2009 PISA reading assessment. Journal of Educational and Behavioral Statistics, 39, 502–523. doi:10.3102/1076998614558485
   Web of Science ®Google Scholar
• Debeer, D., & Janssen, R. (2013). Modeling item-position effects within an IRT framework. Journal of Educational Measurement, 50, 164–185. doi:10.1111/jedm.12009
   Web of Science ®Google Scholar
• Demars, C. E. (2007). Changes in rapid-guessing behavior over a series of assessments. Educational Assessment, 12, 23–45. doi:10.1080/10627190709336946
   Google Scholar
• Duckworth, A. L., & Seligman, M. E. P. (2006). Self-discipline gives girls the edge: Gender in self-discipline, grades, and achievement test scores. Journal of Educational Psychology, 98, 198–208. doi:10.1037/0022-0663.98.1.198
   Web of Science ®Google Scholar
• Enders, C. K., & Tofighi, D. (2007). Centering predictor variables in cross-sectional multilevel models: A new look at an old issue. Psychological Methods, 12, 121–138. doi:10.1037/1082-989X.12.2.121
   PubMed Web of Science ®Google Scholar
• Frey, A., Bernhardt, R., & Born, S. (2017). Umgang mit Itempositionseffekten bei der Entwicklung computerisierter adaptiver Tests [Accounting for item position effects in the development of computerized adaptive tests]. Diagnostica, 63, 167–178. doi:10.1026/0012-1924/a000173
   Google Scholar
• Frey, A., Hartig, J., & Rupp, A. A. (2009). An NCME instructional module on booklet designs in large-scale assessments of student achievement: Theory and practice. Educational Measurement: Issues and Practice, 28, 39–53. doi:10.1111/j.1745-3992.2009.00154.x
   Google Scholar
• Ganzeboom, H. B. G., & Treiman, D. J. (2003). Three internationally standardised measures for comparative research on occupational status. In J. H. P. Hoffmeyer-Zlotnik & C. Wolf (Eds.), Advances in crossnational comparison. A European working book for demographic and socio-economic variables (pp. 159–193). New York, NY: Kluwer Academic/Plenum Publishers.
   Google Scholar
• Hartig, J., & Buchholz, J. (2012). A multilevel item response model for item position effects and individual persistence. Psychological Test and Assessment Modeling, 54, 418–431.
   Google Scholar
• Hohensinn, C., Kubinger, K. D., Reif, M., Schleicher, E., & Khorramdel, L. (2011). Analysing item position effects due to test booklet design within large-scale assessment. Educational Research and Evaluation, 17, 497–509. doi:10.1080/13803611.2011.632668
   Google Scholar
• Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge: Cambridge University Press.
   Google Scholar
• Kreiner, S., & Christensen, K. B. (2014). Analyses of model fit and robustness. A new look at the PISA scaling model underlying ranking of countries according to reading literacy. Psychometrika, 79, 210–231. doi:10.1007/s11336-013-9347-z
   PubMed Web of Science ®Google Scholar
• Leary, L. F., & Dorans, N. J. (1985). Implications for altering the context in which test items appear: A historical perspective on an immediate concern. Review of Educational Research, 55, 387–413. doi:10.3102/00346543055003387
   Web of Science ®Google Scholar
• Lindner, C., Nagy, G., Ramos Arhuis, W. A. R., & Retelsdorf, J. (2017). A new perspective on the interplay between self-control and cognitive performance: Modeling progressive depletion patterns. PLoS One, 12, e0180149. doi:10.1371/journal.pone.0180149
   PubMed Web of Science ®Google Scholar
• List, M. K., Köller, O., & Nagy, G. (2017). A semiparametric approach for modeling not-reached items. Educational and Psychological Measurement, Advanced Online Publication. doi:10.1177/0013164417749679
   Google Scholar
• Maaz, K., Trautwein, U., Ldtke, O., & Baumert, J. (2008). Educational transitions and differential learning environments: How explicit between-school tracking contributes to social inequality in educational outcomes. Child Development Perspectives, 2, 99–106. doi:10.1111/j.1750-8606.2008.00048.x
   Web of Science ®Google Scholar
• Mehta, P. D., & Neale, M. C. (2005). People are variables too: Multilevel structural equations modeling. Psychological Methods, 10, 259–284. doi:10.1037/1082-989X.10.3.259
   PubMed Web of Science ®Google Scholar
• Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–749. doi:10.1037/0003-066X.50.9.741
   Web of Science ®Google Scholar
• Meyers, J. L., Miller, G. E., & Way, W. D. (2009). Item position and item difficulty change in an IRT-based common item equating design. Applied Measurement in Education, 22, 38–60. doi:10.1080/08957340802558342
   Web of Science ®Google Scholar
• Mislevy, R. J., Beaton, A. E., Kaplan, B., & Sheehan, K. M. (1992). Estimating population characteristics from sparse matrix samples of item responses. Journal of Educational Measurement, 29, 133–161. doi:10.1111/j.1745-3984.1992.tb00371.x
   Web of Science ®Google Scholar
• Muthén, L. K., & Muthén, B. O. (2012). Mplus user’s guide (7th ed.). Los Angeles, CA: Muthén & Muthén.
   Google Scholar
• Nagy, G., Haag, N., Oliver, L., & Köller, O. (2017). Längsschnittskalierung der Tests zur Überprüfung des Erreichens der Bildungsstandards der Sekundarstufe I im PISA-Längsschnitt 2012/2013 [Longitudinal IRT scaling of tests of the educational standards for lower secondary level in the PISA longitudinal assessment 2012/2013]. Zeitschrift für Erziehungswissenschaft, 20, 259–286. doi:10.1007/s11618-017-0755-1
   Google Scholar
• Nagy, G., Lüdtke, O., & Köller, O. (2016). Modeling test context effects in longitudinal achievement data: Examining position effects in the longitudinal German PISA 2012 assessment. Psychological Test and Assessment Modeling, 58, 641–670.
   Google Scholar
• Nagy, G., Lüdtke, O., Köller, O., & Heine, J. H. (2017). IRT-Skalierung der Tests im PISA-Längsschnitt 2012/2013: Auswirkungen von Testkontexteffekten auf die Zuwachsschätzung [IRT scaling of the tests in PISA longitudinal assessment 2012/2013: Impact of test context effects on the growth estimate]. Zeitschrift für Erziehungswissenschaft, 20, 229–258. doi:10.1007/s11618-017-0749-z
   Google Scholar
• Nagy, G., Retelsdorf, J., Goldhammer, F., Schiepe-Tiska, A., & Lüdtke, O. (2017). Veränderungen der Lesekompetenz von der 9. zur 10. Klasse: Differenzielle Entwicklungen in Abhängigkeit der Schulform, des Geschlechts und des soziodemografischen Hintergrunds? [Changes in reading skills from 9th to 10th grade: Differential trajectories depending on school type, gender and socio-demographic background?]. Zeitschrift für Erziehungswissenschaft, 20, 177–203. doi:10.1007/s11618-017-0747-1
   Google Scholar
• OECD. (2009). PISA 2006 technical report. Paris: Author.
   Google Scholar
• OECD. (2013). PISA 2012 results: Excellence through equity: Giving every student the chance to succeed (Vol. II). Paris: Author. doi:10.1787/9789264201132-en
   Google Scholar
• Perry, L., & McConney, A. (2010). Does the SES of the school matter? An examination of socioeconomic status and student achievement using PISA 2003. The Teachers College Record, 112, 1137–1162.
   Web of Science ®Google Scholar
• Prenzel, M., Artelt, C., Baumert, J., Blum, W., Hammann, M., Klieme, E., & Pekrun, R. (2008). PISA 2006 in Deutschland – Die Kompetenzen der Jugendlichen im dritten Ländervergleich [PISA 2006 in Germany – The competencies in adolescents in the third state comparison]. Münster: Waxmann.
   Google Scholar
• Qian, J. (2014). An investigation of position effects in large-scale writing assessments. Applied Psychological Measurement, 38, 518–534. doi:10.1177/0146621614534312
   Web of Science ®Google Scholar
• Raudenbush, S., & Willms, J. D. (1995). The estimation of school effects. Journal of Educational and Behavioral Statistics, 20, 307–335. doi:10.3102/10769986020004307
   Web of Science ®Google Scholar
• Ren, X., Goldhammer, F., Moosbrugger, H., & Schweizer, K. (2012). How does attention relate to the ability-specific and position-specific components of reasoning measured by APM? Learning and Individual Differences, 22, 1–7. doi:10.1016/j.lindif.2011.09.009
   Web of Science ®Google Scholar
• Robitzsch, A. (2009). Methodische Herausforderungen bei der Kalibrierung von Leistungstests [Methodological challenges in calibrating achievement tests]. In D. Granzer, O. Köller, A. Bremerich-Vos, M. van den Heuvel-Panhuizen, K. Reiss, & G. Walther (Eds.), Bildungsstandards in Deutsch und Mathematik (pp. 42–107). Weinheim: Beltz.
   Google Scholar
• Rubin, D. B. (1987). Multiple imputation for nonresponse in surveys. Hoboken, NJ: Wiley.10.1002/SERIES1345
   Google Scholar
• Schweizer, K., Troche, S. J., & Rammsayer, T. H. (2011). On the special relationship between fluid and general intelligence: New evidence obtained by considering the position effect. Personality and Individual Differences, 50, 1249–1254. doi:10.1016/j.paid.2011.02.019
   Web of Science ®Google Scholar
• Sirin, S. (2005). Socioeconomic status and academic achievement: A meta-analytic review of research. Review of Educational Research, 75, 417–453. doi:10.3102/00346543075003417
   Web of Science ®Google Scholar
• Warm, T. A. (1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54, 427–450. doi:10.1007/BF02294627.
   Web of Science ®Google Scholar
• Watermann, R., & Klieme, E. (2002). Reporting results of large-scale assessment in psychologically and educationally meaningful terms: Construct validation and proficiency scaling in TIMSS. European Journal of Psychological Assessment, 18, 190–203. doi:10.1027//1015-5759.18.3.190
   Web of Science ®Google Scholar
• Weinert, F. E. (2001). Vergleichende Leistungsmessung in Schulen - eine umstrittene Selbstverständlichkeit. In F. E. Weinert (Ed.), Leistungsmessung in Schulen (pp. 23–43). Weinheim und Basel: Beltz-Verlag.
   Google Scholar
• Weirich, S., Hecht, M., & Böhme, K. (2014). Modeling item position effects using generalized linear mixed models. Applied Psychological Measurement, 38, 535–548. doi:10.1177/0146621614534955
   Web of Science ®Google Scholar
• Weirich, S., Hecht, M., Penk, C., Roppelt, A., & Böhme, K. (2017). Item position effects are moderated by changes in test-taking effort. Applied Psychological Measurement, 41, 115–129. doi:10.1177/0146621616676791
   PubMed Web of Science ®Google Scholar
• Willms, J. D. (2006). Variation in socioeconomic gradients among cantons in French-and Italian-speaking Switzerland: Findings from the OECD PISA. Educational Research and Evaluation, 12, 129–154. doi:10.1080/13803610600587008
   Google Scholar
• Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18, 163–183. doi:10.1207/s15324818ame1802_2
   Web of Science ®Google Scholar
• Wu, M. (2005). The role of plausible values in large-scale surveys. Studies in Educational Evaluation, 31, 114–128. doi:10.1016/j.stueduc.2005.05.005
   Google Scholar
• Wu, M. L., Adams, R. J., Wilson, M. R., & Haldane, S. A. (2007). ACER ConQuest version 2.0: Generalised item response modelling software. Melbourne: ACER Press.
   Google Scholar