Investigating the Measurement of OTL in PISA 2012 and its Relationship with Self-efficacy and Mathematics Achievement: Doubly Latent Multilevel Analyses

References

• Bandura, A. (1997). Self-efficacy: The exercise of control. Freeman.
   Google Scholar
• Barnard-Brak, L., Lan, W. Y., & Yang, Z. (2018). Differences in mathematics achievement according to opportunity to learn: A 4pL item response theory examination. Studies in Educational Evaluation, 56, 1–7. https://doi.org/10.1016/j.stueduc.2017.11.002
   Web of Science ®Google Scholar
• Bliese, P. D. (1998). Group size, ICC values, and group-level correlations: A simulation. Organizational Research Methods, 1(4), 355–373. https://doi.org/10.1177/109442819814001
   Web of Science ®Google Scholar
• Bliese, P. D. (2000). Within group agreement, non-independence and reliability: Implications for data and analysis. In K. J. Klein, & S. W. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 349–381). Jossey-Bass.
   Google Scholar
• Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111(4), 1061–1071. https://doi.org/10.1037/0033-295X.111.4.1061
   PubMed Web of Science ®Google Scholar
• Carroll, J. B. (1963). A model of school learning. Teachers College Record: The Voice of Scholarship in Education, 64(8), 1–9. https://doi.org/10.1177/016146816306400801
   Web of Science ®Google Scholar
• Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 464–504. https://doi.org/10.1080/10705510701301834
   Web of Science ®Google Scholar
• Chen, Y. F., & Jiao, H. (2014). Exploring the utility of background and cognitive variables in explaining latent differential item functioning: An example of the PISA 2009 reading assessment. Educational Assessment, 19(2), 77–96. https://doi.org/10.1080/10627197.2014.903650
   Google Scholar
• Chmielewski, A. K. (2014). An international comparison of achievement inequality in within- and between-school tracking systems. American Journal of Education, 120(3), 293–324. https://doi.org/10.1086/675529
   Web of Science ®Google Scholar
• Chou, C. P., Bentler, P. M., & Satorra, A. (1991). Scaled test statistics and robust standard errors for non-normal data in covariance structure analysis: A monte carlo study. British Journal of Mathematical and Statistical Psychology, 44(2), 347–357. https://doi.org/10.1111/j.2044-8317.1991.tb00966.x
   PubMed Web of Science ®Google Scholar
• Cogan, L. S., & Schmidt, W. H. (2015). The concept of opportunity to learn (OTL) in international comparisons of education. In K. Stacey & R. Turner (Eds.), Assessing mathematical literacy: The PISA experience (pp. 207–216). Springer.
   Google Scholar
• Cogan, L. S., Schmidt, W. H., & Guo, S. (2019). The role that mathematics plays in college- and career-readiness: Evidence from PISA. Journal of Curriculum Studies, 51(4), 530–553. https://doi.org/10.1080/00220272.2018.1533998
   Web of Science ®Google Scholar
• Cohen, A., Doveh, E., & Eick, U. (2001). Statistical properties of the rWG(J) index of agreement. Psychological Methods, 6(3), 297–310. https://doi.org/10.1037/1082-989X.6.3.297
   PubMed Web of Science ®Google Scholar
• Coleman, J. S. (1966). The concept of equality of educational opportunity. US Office of Education.
   Google Scholar
• Connor, C. M., Morrison, F. J., Fishman, B. J., Ponitz, C. C., Glasney, S., Underwood, P. S., Piasta, S. B., Crowe, E. C., & Schatschneider, C. (2009). The ISI classroom observation system: Examining the literacy instruction provided to individual students. Educational Researcher, 38(2), 85–99. https://doi.org/10.3102/0013189X09332373
   Web of Science ®Google Scholar
• Enders, C. K. (2010). Applied missing data analysis (Methodology in the social sciences). Guilford Publications.
   Google Scholar
• Floden, R. (2002). The measurement of opportunity to learn. In A. Porter, & A. Gamoran (Eds.), Methodological advances in cross-national surveys of educational achievement (pp. 231–266). National Academy Press.
   Google Scholar
• Foldnes, N., & Grønneberg, S. (2020). Pernicious polychorics: The impact and detection of underlying non-normality. Structural Equation Modeling: A Multidisciplinary Journal, 27(4), 525–543. https://doi.org/10.1080/10705511.2019.1673168
   Web of Science ®Google Scholar
• Foldnes, N., & Grønneberg, S. (2021). The sensitivity of structural equation modeling with ordinal data to underlying non-normality and observed distributional forms. Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000385
   PubMed Web of Science ®Google Scholar
• Fox, R. (2001). Constructivism examined. Oxford Review of Education, 27(1), 23–35. https://doi.org/10.1080/03054980125310
   Web of Science ®Google Scholar
• Gee, J. P. (2008). A sociocultural perspective on opportunity to learn. In P. A. Moss, D. C. Pullin, J. P. Gee, E. H. Haertel, & L. J. Young (Eds.), Assessment, equity, and opportunity to learn (pp. 76–108). Cambridge University Press.
   Google Scholar
• Hansen, K. Y., & Strietholt, R. (2018). Does schooling actually perpetuate educational inequality in mathematics performance? A validity question on the measures of opportunity to learn in PISA. ZDM, 50(4), 643–658. https://doi.org/10.1007/s11858-018-0935-3
   Google Scholar
• He, J., Barrera-Pedemonte, F., & Buchholz, J. (2019). Cross-cultural comparability of noncognitive constructs in TIMSS and PISA. Assessment in Education: Principles, Policy & Practice, 26(4), 369–385. https://doi.org/10.1080/0969594X.2018.1469467
   Web of Science ®Google Scholar
• Honicke, T., & Broadbent, J. (2016). The influence of academic self-efficacy on academic performance: A systematic review. Educational Research Review, 17, 63–84. https://doi.org/10.1016/j.edurev.2015.11.002
   Web of Science ®Google Scholar
• Houang, R. T., & Schmidt, W. H. (2008). TIMSS international curriculum analysis andmeasuring educational opportunities. 3rd IEA international research conference. http://www.iea.nl/fileadmin/user_upload/IRC/IRC_2008/Papers/IRC2008_Houang_Schmidt.pdf
   Google Scholar
• Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
   Web of Science ®Google Scholar
• Husén, T. (1967). International study of achievement in mathematics: A comparison of twelve countries. Wiley.
   Google Scholar
• Irvin, M., Byun, S. Y., Smiley, W. S., & Hutchins, B. C. (2017). Relation of opportunity to learn advanced math to the educational attainment of rural youth. American Journal of Education, 123(3), 475–510. https://doi.org/10.1086/691231
   PubMed Web of Science ®Google Scholar
• Jia, Y., & Konold, T. (2019). Moving to the next level: Doubly latent multilevel mediation models with a school climate illustration. The Journal of Experimental Education, 89(2), 422–440. https://doi.org/10.1080/00220973.2019.1675136
   Web of Science ®Google Scholar
• Kline, R. B. (2005). Principles and practices of structural equation modeling. The Guildford Press.
   Google Scholar
• Lafontaine, D., Baye, A., Vieluf, S., & Monseur, C. (2015). Equity in opportunity-to-learn and achievement in reading: A secondary analysis of PISA 2009 data. Studies in Educational Evaluation, 47, 1–11. https://doi.org/10.1016/j.stueduc.2015.05.001
   Web of Science ®Google Scholar
• Lam, A. C., Ruzek, E. A., Schenke, K., Conley, A. M., & Karabenick, S. A. (2015). Student perceptions of classroom achievement goal structure: Is it appropriate to aggregate? Journal of Educational Psychology, 107(4), 1102–1115. https://doi.org/10.1037/edu0000028
   Web of Science ®Google Scholar
• Laukaityte, I., & Wiberg, M. (2017). Using plausible values in secondary analysis in large-scale assessments. Communications in Statistics - Theory and Methods, 46(22), 11341–11357. https://doi.org/10.1080/03610926.2016.1267764
   Web of Science ®Google Scholar
• LeBreton, J. M., & Senter, J. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11(4), 815–852. https://doi.org/10.1177/1094428106296642
   Web of Science ®Google Scholar
• Lee, J., & Stankov, L. (2013). Higher-order structure of noncognitive constructs and prediction of PISA 2003 mathematics achievement. Learning and Individual Differences, 26, 119–130. https://doi.org/10.1016/j.lindif.2013.05.004
   Web of Science ®Google Scholar
• Lee, J., & Stankov, L. (2018). Non-cognitive predictors of academic achievement: Evidence from TIMSS and PISA. Learning and Individual Differences, 65, 50–64. https://doi.org/10.1016/j.lindif.2018.05.009
   Web of Science ®Google Scholar
• Lee, W., Lee, M. J., & Bong, M. (2014). Testing interest and self-efficacy as predictors of academic self-regulation and achievement. Contemporary Educational Psychology, 39(2), 86–99. https://doi.org/10.1016/j.cedpsych.2014.02.002
   Web of Science ®Google Scholar
• Lewis, R. W., & Farkas, G. (2017). Using an opportunity-propensity framework to estimate individual-, classroom-, and school-level predictors of middle school science achievement. Contemporary Educational Psychology, 51, 185–197. https://doi.org/10.1016/j.cedpsych.2017.08.003
   Web of Science ®Google Scholar
• Lüdtke, O., Marsh, H. W., Robitzsch, A., & Trautwein, U. (2011). A 2 × 2 taxonomy of multilevel latent contextual models: Accuracy-bias trade-offs in full and partial error correction models. Psychological Methods, 16(4), 444–467. https://doi.org/10.1037/a0024376
   PubMed Web of Science ®Google Scholar
• Lüdtke, O., Marsh, H. W., Robitzsch, A., Trautwein, U., Asparouhov, T., & Muthén, B. (2008). The multilevel latent covariate model: A new, more reliable approach to group-level effects in contextual studies. Psychological Methods, 13(3), 203–229. https://doi.org/10.1037/a0012869
   PubMed Web of Science ®Google Scholar
• Marsh, H. W., Hau, K. T., & Wen, Z. (2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for Fit indexes and dangers in overgeneralizing Hu and Bentler’s (1999) findings. Structural Equation Modeling: A Multidisciplinary Journal, 11(3), 320–341. https://doi.org/10.1207/s15328007sem1103_2
   Web of Science ®Google Scholar
• Marsh, H. W., Lüdtke, O., Nagengast, B., Trautwein, U., Morin, A. J., Abduljabbar, A. S., & Köller, O. (2012). Classroom climate and contextual effects: Conceptual and methodological issues in the evaluation of group-level effects. Educational Psychologist, 47(2), 106–124. https://doi.org/10.1080/00461520.2012.670488
   Web of Science ®Google Scholar
• Marsh, H. W., Lüdtke, O., Robitzsch, A., Trautwein, U., Asparouhov, T., Muthén, B., & Nagengast, B. (2009). Doubly-latent models of school contextual effects: Integrating multilevel and structural equation approaches to control measurement and sampling error. Multivariate Behavioral Research, 44(6), 764–802. https://doi.org/10.1080/00273170903333665
   PubMed Web of Science ®Google Scholar
• McDonnell, L. M. (1995). Opportunity to learn as a research concept and a policy instrument. Educational Evaluation and Policy Analysis, 17(3), 305–322. https://doi.org/10.3102/01623737017003305
   Web of Science ®Google Scholar
• Mehta, P. D., & Neale, M. C. (2005). People are variables too: Multilevel structural equations modeling. Psychological Methods, 10(3), 259–284. https://doi.org/10.1037/1082-989X.10.3.259
   PubMed Web of Science ®Google Scholar
• Minor, E. C., Desimone, L. M., Phillips, K. J. R., & Spencer, K. (2015). A new look at the opportunity-to-learn gap across race and income. American Journal of Education, 121(2), 241–269. https://doi.org/10.1086/679392
   Web of Science ®Google Scholar
• Mohammadpour, E., & Abdul Ghafar, M. N. (2014). Mathematics achievement as a function of within- and between-school differences. Scandinavian Journal of Educational Research, 58(2), 189–221. https://doi.org/10.1080/00313831.2012.725097
   Web of Science ®Google Scholar
• Morin, A. J., Marsh, H. W., Nagengast, B., & Scalas, L. F. (2014). Doubly latent multilevel analyses of classroom climate: An illustration. The Journal of Experimental Education, 82(2), 143–167. https://doi.org/10.1080/00220973.2013.769412
   Web of Science ®Google Scholar
• Muthén, B. O., & Satorra, A. (1995). Complex sample data in structural equation modeling. Sociological Methodology, 25, 267–316. https://doi.org/10.2307/271070
   Web of Science ®Google Scholar
• Muthén, L. K., & Muthén, B. O. (2017). Mplus user’s guide. Muthén & Muthén.
   Google Scholar
• OECD. (2009). PISA data analysis manual: SPSS (2nd ed.). OECD Publishing. https://doi.org/10.1787/9789264056275-en
   Google Scholar
• OECD. (2013). PISA 2012 assessment and analytical framework: Mathematics, reading, science, problem solving and financial literacy. OECD Publishing. https://doi.org/10.1787/9789264190511-en
   Google Scholar
• OECD. (2014a). PISA 2012 technical report. OECD Publishing.
   Google Scholar
• OECD. (2014b). PISA 2012 results: What students know and can do – student performance in mathematics, reading and science (Volume I). OECD Publishing. https://doi.org/10.1787/9789264208780-en
   Google Scholar
• Posel, D., & Grapsa, E. (2017). Time to learn? Time allocations among children in South Africa. International Journal of Educational Development, 56, 1–10. https://doi.org/10.1016/j.ijedudev.2017.07.002
   Web of Science ®Google Scholar
• Preacher, K. J. (2015). Advances in mediation analysis: A survey and synthesis of new developments. Annual Review of Psychology, 66(1), 825–852. https://doi.org/10.1146/annurev-psych-010814-015258
   PubMed Web of Science ®Google Scholar
• Preacher, K. J., Zhang, Z., & Zyphur, M. J. (2011). Alternative methods for assessing mediation in multilevel data: The advantages of multilevel SEM. Structural Equation Modeling: A Multidisciplinary Journal, 18(2), 161–182. https://doi.org/10.1080/10705511.2011.557329
   Web of Science ®Google Scholar
• Preacher, K. J., Zyphur, M. J., & Zhang, Z. (2010). A general multilevel SEM framework for assessing multilevel mediation. Psychological Methods, 15(3), 209–233. https://doi.org/10.1037/a0020141
   PubMed Web of Science ®Google Scholar
• Reeves, C., Carnoy, M., & Addy, N. (2013). Comparing opportunity to learn and student achievement gains in southern African primary schools: A new approach. International Journal of Educational Development, 33(5), 426–435. https://doi.org/10.1016/j.ijedudev.2012.12.006
   Web of Science ®Google Scholar
• Richardson, M., Abraham, C., & Bond, R. (2012). Psychological correlates of university students’ academic performance: A systematic review and meta-analysis. Psychological Bulletin, 138(2), 353–387. https://doi.org/10.1037/a0026838
   PubMed Web of Science ®Google Scholar
• Rolfe, V., Strietholt, R., & Hansen, K. Y. (2021). Does inequality in opportunity perpetuate inequality in outcomes? International evidence from four TIMSS cycles. Studies in Educational Evaluation, 71, Article 101086. https://doi.org/10.1016/j.stueduc.2021.101086
   Web of Science ®Google Scholar
• Rubin, D. B. (2004). Multiple imputation for nonresponse in surveys. John Wiley & Sons.
   Google Scholar
• Santibañez, L., & Fagioli, L. (2016). Nothing succeeds like success? Equity, student outcomes, and opportunity to learn in high- and middle-income countries. International Journal of Behavioral Development, 40(6), 517–525. https://doi.org/10.1177/0165025416642050
   Web of Science ®Google Scholar
• Scheerens, J. (2016). Meta-analyses and descriptions of illustrative studies. In J. Scheerens (Ed.), Opportunity to learn, instructional alignment and test preparation: A research review (pp. 28–60). University of Oslo.
   Google Scholar
• Schlomer, G. L., Bauman, S., & Card, N. A. (2010). Best practices for missing data management in counseling psychology. Journal of Counseling Psychology, 57(1), 1–10. https://doi.org/10.1037/a0018082
   PubMed Web of Science ®Google Scholar
• Schmidt, W. H., & Burroughs, N. A. (2013). Opening the black box: Prospects for using international large-scale assessments to explore classroom effects. Research in Comparative and International Education, 8(3), 236–247. https://doi.org/10.2304/rcie.2013.8.3.236
   Google Scholar
• Schmidt, W. H., Burroughs, N. A., Zoido, P., & Houang, R. T. (2015). The role of schooling in perpetuating educational inequality. Educational Researcher, 44(7), 371–386. https://doi.org/10.3102/0013189X15603982
   Web of Science ®Google Scholar
• Schmidt, W. H., Cogan, L. S., Houang, R. T., & McKnight, C. C. (2011). Content coverage differences across districts/states: A persisting challenge for U.S. education policy. American Journal of Education, 117(3), 399–427. https://doi.org/10.1086/659213
   Web of Science ®Google Scholar
• Schmidt, W. H., Guo, S., & Houang, R. T. (2021). The role of opportunity to learn in ethnic inequality in mathematics. Journal of Curriculum Studies, 579–600. https://doi.org/10.1080/00220272.2020.1863475
   Web of Science ®Google Scholar
• Schmidt, W. H., & McKnight, C. C. (1995). Surveying educational opportunity in mathematics and science: An international perspective. Educational Evaluation and Policy Analysis, 17(3), 337–353. https://doi.org/10.3102/01623737017003337
   Web of Science ®Google Scholar
• Schmidt, W. H., McKnight, C. C., Cogan, L. S., Jakwerth, P. M., & Houang, R. T. (1999). Facing the consequences: Using TIMSS for a closer look at US mathematics and science education. Kluwer.
   Google Scholar
• Schmidt, W. H., McKnight, C. C., Houang, R. T., Wang, H. A., Wiley, D. E., Cogan, L. S., & Wolfe, R. G. (2001). Why schools matter: A cross-national comparison of curriculum and learning. Jossey-Bass.
   Google Scholar
• Schmidt, W. H., Zoido, P., & Cogan, L. S. (2014). Schooling matters: Opportunity to learn in PISA 2012 (OECD Education Working Papers No. 95). OECD Publishing. https://doi.org/10.1787/5k3v0hldmchl-en
   Google Scholar
• Schnell, K., Ringeisen, T., Raufelder, D., & Rohrmann, S. (2015). The impact of adolescents’ self-efficacy and self-regulated goal attainment processes on school performance — do gender and test anxiety matter? Learning and Individual Differences, 38, 90–98. https://doi.org/10.1016/j.lindif.2014.12.008
   Web of Science ®Google Scholar
• Schunk, D. H. (1989). Self-efficacy and achievement behaviors. Educational Psychology Review, 1(3), 173–208. https://doi.org/10.1007/BF01320134
   Web of Science ®Google Scholar
• Schunk, D. H., & Pajares, F. (2009). Self-efficacy theory. In K. R. Wentzel, & D. B. Miele (Eds.), Handbook of motivation at school (pp. 35–44). Routledge.
   Google Scholar
• Skaalvik, E. M., Federici, R. A., & Klassen, R. M. (2015). Mathematics achievement and self-efficacy: Relations with motivation for mathematics. International Journal of Educational Research, 72, 129–136. https://doi.org/10.1016/j.ijer.2015.06.008
   Web of Science ®Google Scholar
• Stevens, F. I. (1993). Applying an opportunity-to-learn conceptual framework to the investigation of the effects of teaching practices via secondary analyses of multiple- case-study summary data. The Journal of Negro Education, 62(3), 232–248. https://doi.org/10.2307/2295463
   Google Scholar
• Sungur, S. (2007). Modeling the relationships among students’ motivational beliefs, metacognitive strategy use, and effort regulation. Scandinavian Journal of Educational Research, 51(3), 315–326. https://doi.org/10.1080/00313830701356166
   Web of Science ®Google Scholar
• Talsma, K., Schüz, B., Schwarzer, R., & Norris, K. (2018). I believe, therefore I achieve (and vice versa): A meta-analytic cross-lagged panel analysis of self-efficacy and academic performance. Learning and Individual Differences, 61, 136–150. https://doi.org/10.1016/j.lindif.2017.11.015
   Web of Science ®Google Scholar
• Urick, A., Wilson, A. S., Ford, T. G., Frick, W. C., & Wronowski, M. L. (2018). Testing a framework of math progress indicators for ESSA: How opportunity to learn and instructional leadership matter. Educational Administration Quarterly, 54(3), 396–438. https://doi.org/10.1177/0013161X18761343
   Web of Science ®Google Scholar
• Usher, E. L., Li, C. R., Butz, A. R., & Rojas, J. P. (2019). Perseverant grit and self-efficacy: Are both essential for children’s academic success? Journal of Educational Psychology, 111(5), 877–902. https://doi.org/10.1037/edu0000324
   Web of Science ®Google Scholar
• Wang, F., Liu, Y., & Leung, S. O. (2022). Disciplinary climate, opportunity to learn, and mathematics achievement: An analysis using doubly latent multilevel structural equation modeling. School Effectiveness and School Improvement, 1–18. https://doi.org/10.1080/09243453.2022.2043393
   Web of Science ®Google Scholar
• Wang, J. (1998). Opportunity to learn: The impacts and policy implications. Educational Evaluation and Policy Analysis, 20(3), 137–156. https://doi.org/10.3102/01623737020003137
   Web of Science ®Google Scholar
• Wiley, D. E., & Harnischfeger, A. (1974). Explosion of a myth: Quantity of schooling and exposure to instruction, major educational vehicles. Educational Researcher, 3(4), 7–12. https://doi.org/10.3102/0013189X003004007
   Google Scholar
• Woehr, D. J., Loignon, A. C., Schmidt, P. B., Loughry, M. L., & Ohland, M. W. (2015). Justifying aggregation with consensus-based constructs: A review and examination of cutoff values for common aggregation indices. Organizational Research Methods, 18(4), 704–737. https://doi.org/10.1177/1094428115582090
   Web of Science ®Google Scholar
• Yeung, S. S., King, R. B., Nalipay, M. J. N., & Cai, Y. (2022). Exploring the interplay between socioeconomic status and reading achievement: An expectancy-value perspective. British Journal of Educational Psychology, e12495. https://doi.org/10.1111/bjep.12495
   Web of Science ®Google Scholar
• Zimmerman, B. J., & Bandura, A. (1994). Impact of self-regulatory influences on writing course attainment. American Educational Research Journal, 31(4), 845–862. https://doi.org/10.3102/00028312031004845
   Web of Science ®Google Scholar
• Zyphur, M. J., Kaplan, S. A., & Christian, M. S. (2008). Assumptions of cross-level measurement and structural invariance in the analysis of multilevel data: Problems and solutions. Group Dynamics: Theory, Research, and Practice, 12(2), 127–140. https://doi.org/10.1037/1089-2699.12.2.127
   Web of Science ®Google Scholar