Exploring internal structure of a performance-based critical thinking assessment for new students in higher education

References

• Aloisi, C., and A. Callaghan. 2018. “Threats to the Validity of the Collegiate Learning Assessment (CLA+) as a Measure of Critical Thinking Skills and Implications for Learning Gain.” Higher Education Pedagogies 3 (1): 57–82. doi:https://doi.org/10.1080/23752696.2018.1449128.
   Web of Science ®Google Scholar
• American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. 2014. Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association.
   Google Scholar
• Andrews, R. 2015. “Critical Thinking and/or Argumentation in Higher Education.” In The Palgrave Handbook of Critical Thinking in Higher Education, edited by M. Davies and D. Barnett, 49–62. New York: Palgrave Macmillan.
   Google Scholar
• Arum, R., and J. Roksa. 2011. Academically Adrift: Limited Learning on College Campuses. Chicago: University of Chicago Press.
   Google Scholar
• Badcock, P., P. Pattison, and K. L. Harris. 2010. “Developing Generic Skills through University Study: A Study of Arts, Science and Engineering in Australia.” Higher Education 60 (4): 441–458. doi:https://doi.org/10.1007/s10734-010-9308-8.
   Web of Science ®Google Scholar
• Beckman, K., T. Apps, S. Bennett, B. Dalgarno, G. Kennedy, and L. Lockyer. 2021. “Self-Regulation in Open-Ended Online Assignment Tasks: The Importance of Initial Task Interpretation and Goal Setting.” Studies in Higher Education 46 (4): 815–821. doi:https://doi.org/10.1080/03075079.2019.1654450.
   Web of Science ®Google Scholar
• Borowiec, K., and C. Castle. 2019. “Using Rater Cognition to Improve Generalizability of an Assessment of Scientific Argumentation.” Practical Assessment, Research, and Evaluation 24 (1): 8.
   Google Scholar
• Bowman, N. A. 2010. “Can 1st-Year College Students Accurately Report Their Learning and Development?” American Educational Research Journal 47 (2): 466–496. doi:https://doi.org/10.3102/0002831209353595.
   Web of Science ®Google Scholar
• Costello, A., and J. Osborne. 2005. “Best Practices in Exploratory Factor Analysis: Four Recommendations for Getting the Most from Your Analysis.” Practical Assessment, Research, and Evaluation 10 (1): 7.
   Google Scholar
• Davey, T., S. Ferrara, R. Shavelson, P. Holland, N. Webb, and L. Wise. 2015. Psychometric Considerations for the Next Generation of Performance Assessment. Washington, DC: Center for K-12 Assessment & Performance Management, Educational Testing Service.
   Google Scholar
• Ennis, R. 1991. “Critical Thinking: A Streamlined Conception.” Teaching Philosophy 14 (1): 5–24. doi:https://doi.org/10.5840/teachphil19911412.
   Google Scholar
• Evens, M., A. Verburgh, and J. Elen. 2013. “Critical Thinking in College Freshmen: The Impact of Secondary and Higher Education.” International Journal of Higher Education 2 (3): 139–151. doi:https://doi.org/10.5430/ijhe.v2n3p139.
   Google Scholar
• Fabrigar, L., D. Wegener, R. MacCallum, and E. Strahan. 1999. “Evaluating the Use of Exploratory Factor Analysis in Psychological Research.” Psychological Methods 4 (3): 272–299. doi:https://doi.org/10.1037/1082-989X.4.3.272.
   Web of Science ®Google Scholar
• Facione, P. 1990. Critical Thinking: A Statement of Expert Consensus for Purposes of Educational Assessment and Instruction. Newark: The American Philosophical Association.
   Google Scholar
• Finnish National Board on Research Integrity. 2019. The Ethical Principles of Research with Human Participants and Ethical Review in the Human Sciences in Finland. Helsinki: Finnish National Board on Research Integrity TENK.
   Google Scholar
• Fischer, F., C. Chinn, K. Engelmann, and J. Osborne, eds. 2018. Scientific Reasoning and Argumentation: The Roles of Domain-Specific and Domain-General Knowledge. New York: Routledge.
   Google Scholar
• Goretzko, D., T. Pham, and M. Bühner. 2019. “Exploratory Factor Analysis: Current Use, Methodological Developments and Recommendations for Good Practice.” Current Psychology 1–12.
   Web of Science ®Google Scholar
• Halpern, D. 2014. Thought and Knowledge: An Introduction to Critical Thinking. 5th ed. New York, London: Psychology Press.
   Google Scholar
• Hambleton, R., P. Merenda, and C. Spielberger. 2005. Adapting Educational and Psychological Tests for Cross-Cultural Assessment. Mahwah, NJ: Psychology Press.
   Google Scholar
• Heijltjes, A., T. van Gog, J. Leppink, and F. Paas. 2014. “Improving Critical Thinking: Effects of Dispositions and Instructions on Economics Students’ Reasoning Skills.” Learning and Instruction 29: 31–42. doi:https://doi.org/10.1016/j.learninstruc.2013.07.003.
   Web of Science ®Google Scholar
• Heiskala, L., J. Erola, and E. Kilpi-Jakonen. 2020. “Compensatory and Multiplicative Advantages: Social Origin, School Performance, and Stratified Higher Education Enrolment in Finland.” European Sociological Review 37 (2):1–15.
   Web of Science ®Google Scholar
• Hetmanek, A., K. Engelmann, A. Opitz, and F. Fischer. 2018. “Beyond Intelligence and Domain Knowledge.” In Scientific Reasoning and Argumentation – The Roles of Domain-Specific and Domain-General Knowledge, edited byF. Fischer, C. Chinn, K. Engelmann, and J. Osborne, 205–226. New York: Routledge.
   Google Scholar
• Hu, L., and P. Bentler. 1998. “Fit Indices in Covariance Structure Modeling: Sensitivity to Underparameterized Model Misspecification.” Psychological Methods 3 (4): 424–453. doi:https://doi.org/10.1037/1082-989X.3.4.424.
   Web of Science ®Google Scholar
• Hu, L., and P. Bentler. 1999. “Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria versus New Alternatives.” Structural Equation Modeling: A Multidisciplinary Journal 6 (1): 1–55. doi:https://doi.org/10.1080/10705519909540118.
   Web of Science ®Google Scholar
• Hyytinen, H., K. Holma, A. Toom, R. Shavelson, and S. Lindblom-Ylänne. 2014. “The Complex Relationship between Students’ Critical Thinking and Epistemological Beliefs in the Context of Problem Solving.” Frontline Learning Research 2 (5): 1–25.
   Google Scholar
• Hyytinen, H., K. Nissinen, J. Ursin, A. Toom, and S. Lindblom-Ylänne. 2015. “Problematising the Equivalence of the Test Results of Performance-Based Critical Thinking Tests for Undergraduate Students.” Studies in Educational Evaluation 44: 1–8. doi:https://doi.org/10.1016/j.stueduc.2014.11.001.
   Web of Science ®Google Scholar
• Hyytinen, H., and A. Toom. 2019. “Developing a Performance Assessment Task in the Finnish Higher Education Context: Conceptual and Empirical Insights.” The British Journal of Educational Psychology 89 (3): 551–563. doi:https://doi.org/10.1111/bjep.12283.
   PubMed Web of Science ®Google Scholar
• Hyytinen, H., A. Toom, and R. Shavelson. 2019. “Enhancing Scientific Thinking through the Development of Critical Thinking in Higher Education.” In Redefining Scientific Thinking for Higher Education, edited by M. Murtonen and K. Balloo, 59–78. Cham: Palgrave Macmillan.
   Google Scholar
• Hyytinen, H., J. Ursin, K. Silvennoinen, K. Kleemola, and A. Toom. 2020. “The Dynamic Relationship between Cognitive Response Processes and Self-Regulation of Cognition in Critical Thinking Assessments.” Manuscript submitted for publication.
   Google Scholar
• International Test Commission. 2018. “ITC Guidelines for Translating and Adapting Tests (Second Edition).” International Journal of Testing 18 (2): 101–134.
   Web of Science ®Google Scholar
• Kleemola, K., and H. Hyytinen. 2019. “Exploring the Relationship between Law Students’ Prior Performance and Academic Achievement at University.” Education Sciences 9 (3): 236. doi:https://doi.org/10.3390/educsci9030236.
   Web of Science ®Google Scholar
• Klein, S., R. Benjamin, R. Shavelson, and R. Bolus. 2007. “The Collegiate Learning Assessment: Facts and Fantasies.” Evaluation Review 31 (5): 415–439. doi:https://doi.org/10.1177/0193841X07303318.
   PubMed Web of Science ®Google Scholar
• Kuhn, D. 2019. “Critical Thinking as Discourse.” Human Development 62 (3): 146–164. doi:https://doi.org/10.1159/000500171.
   Web of Science ®Google Scholar
• Leighton, J. 2019. “The Risk–Return Trade‐off: Performance Assessments and Cognitive Validation of Inferences.” British Journal of Educational Psychology 89 (3): 441–455. doi:https://doi.org/10.1111/bjep.12271.
   PubMed Web of Science ®Google Scholar
• McClelland, D. 1973. “Testing for Competence Rather than for ‘intelligence’.” The American Psychologist 28 (1): 1–14. doi:https://doi.org/10.1037/h0034092.
   PubMed Web of Science ®Google Scholar
• Meade, A., E. Johnson, and P. Braddy. 2008. “Power and Sensitivity of Alternative Fit Indices in Tests of Measurement Invariance.” The Journal of Applied Psychology 93 (3): 568–592. doi:https://doi.org/10.1037/0021-9010.93.3.568.
   PubMed Web of Science ®Google Scholar
• Messick, S. 1994. Alternative Modes of Assessment, Uniform Standards of Validity. Princeton: Educational Testing Service.
   Google Scholar
• Messick, S. 1995. “Validity of Psychological Assessment: Validation of Inferences from Persons’ Responses and Performances as Scientific Inquiry into Score Meaning.” American Psychologist 50 (9): 741.
   Web of Science ®Google Scholar
• Muthén, L., and B. Muthén. 2017. Mplus User’s Guide. 8th ed. Los Angeles: Muthén & Muthén.
   Google Scholar
• O’Connor, B. 2000. “SPSS and SAS Programs for Determining the Number of Components Using Parallel Analysis and Velicer’s MAP Test.” Behavior Research Methods, Instruments, & Computers 32 (3): 396–402. doi:https://doi.org/10.3758/BF03200807.
   PubMedGoogle Scholar
• O’Hare, L., and C. McGuinness. 2015. “The Validity of Critical Thinking Tests for Predicting Degree Performance: A Longitudinal Study.” International Journal of Educational Research 72: 162–172. doi:https://doi.org/10.1016/j.ijer.2015.06.004.
   Web of Science ®Google Scholar
• Shavelson, R. 2010. Measuring College Learning Responsibly: Accountability in a New Era. Stanford: Stanford University Press.
   Google Scholar
• Shavelson, R. 2018. “Discussion of Papers andReflections on ‘Exploring the Limits of Domain-Generality’.” In Scientific Reasoning and Argumentation: The Roles of Domain-Specific and Domain-General Knowledge, edited by F. Fischer, C. Chinn, K. Engelmann, and J. Osborne, 112–118. New York: Routledge.
   Google Scholar
• Shavelson, R., G. Baxter, and X. Gao. 1993. “Sampling Variability of Performance Assessments.” Journal of Educational Measurement 30 (3): 215–232. doi:https://doi.org/10.1111/j.1745-3984.1993.tb00424.x.
   Web of Science ®Google Scholar
• Shavelson, R., O. Zlatkin-Troitschanskaia, K. Beck, S. Schmidt, and J. Marino. 2019. “Assessment of University Students’ Critical Thinking: Next Generation Performance Assessment.” International Journal of Testing 19 (4):‌ 1–20.
   Web of Science ®Google Scholar
• Shavelson, R., O. Zlatkin-Troitschanskaia, and J. Marino. 2018. “International Performance Assessment of Learning in Higher Education (iPAL): Research and Development.” In Assessment of Learning Outcomes in Higher Education, edited by O. Zlatkin-Troitschanskaia, M. Toepper, H. Pant, C. Lautenbach, and C. Kuhn, 193–214. Cham: Springer.
   Google Scholar
• Solano-Flores, G., and M. Chía. 2017. “Validation of Score Meaning in Multiple Language Versions of Tests.” In Validation of Score Meaning for the Next Generation of Assessments: The Use of Response Processes, edited by K. Ercikan and J. Pellegrino, 127–137. New York: Routledge.
   Google Scholar
• Tabachnick, B., and L. Fidell. 2014. Using Multivariate Statistics. 6th ed. Harlow: Pearson.
   Google Scholar
• Tuononen, T., A. Parpala, and S. Lindblom-Ylänne. 2019. “Graduates’ Evaluations of Usefulness of University Education, and Early Career Success – A Longitudinal Study of the Transition to Working Life.” Assessment & Evaluation in Higher Education 44 (4):‌ 1–14.
   Web of Science ®Google Scholar
• van der Zanden, P., E. Denessen, A. Cillessen, and P. Meijer. 2019. “Patterns of Success: First-Year Student Success in Multiple Domains.” Studies in Higher Education 44 (11): 2081–2095. doi:https://doi.org/10.1080/03075079.2018.1493097.
   Web of Science ®Google Scholar
• Vandenberg, R., and C. Lance. 2000. “A Review and Synthesis of the Measurement Invariance Literature: Suggestions, Practices, and Recommendations for Organizational Research.” Organizational Research Methods 3 (1): 4–70. doi:https://doi.org/10.1177/109442810031002.
   Web of Science ®Google Scholar
• Zahner, D. 2013. Reliability and Validity CLA +. New York: Council for Aid to Education.
   Google Scholar
• Zahner, D., and A. Ciolfi. 2018. “International Comparison of a Performance-Based Assessment in Higher Education.” In Assessment of Learning Outcomes in Higher Education, edited by O. Zlatkin-Troitschanskaia, M. Toepper, H. Pant, C. Lautenbach, and C. Kuhn, 215–244. Cham: Springer.
   Google Scholar
• Zlatkin-Troitschanskaia, O., R. Shavelson, and C. Kuhn. 2015. “The International State of Research on Measurement of Competency in Higher Education.” Studies in Higher Education 40 (3): 393–411. doi:https://doi.org/10.1080/03075079.2015.1004241.
   Web of Science ®Google Scholar
• Zlatkin-Troitschanskaia, O., R. Shavelson, S. Schmidt, and K. Beck. 2019. “On the Complementarity of Holistic and Analytic Approaches to Performance Assessment Scoring.” The British Journal of Educational Psychology 89 (3): 468–484. doi:https://doi.org/10.1111/bjep.12286.
   PubMed Web of Science ®Google Scholar