3,559
Views
9
CrossRef citations to date
0
Altmetric
Articles

Threats to the validity of the Collegiate Learning Assessment (CLA+) as a measure of critical thinking skills and implications for Learning Gain

ORCID Icon & ORCID Icon
Pages 57-82 | Received 02 Oct 2017, Accepted 18 Feb 2018, Published online: 06 Sep 2018

References

  • ACT . (2017). ACT collegiate assessment of academic proficiency [Web page]. Retrieved from http://www.act.org/content/act/en/products-and-services/act-collegiate-assessment-of-academic-proficiency.html
  • Alexander, P.A. (2014). Thinking critically and analytically about critical-analytic thinking: An introduction. Educational Psychology Review , 26 (4), 469–476. doi:10.1007/s10648-014-9283-1
  • Arum, R. (2013). Stakeholder and public responses to measuring student learning. Society , 50 (3), 230–235. doi:10.1007/s12115-013-9648-y
  • Arum, R. , Cho, E. , Kim, J. , & Roksa, J. (2012). Documenting uncertain times: Post-graduate transitions of the Academically Adrift cohort . New York, NY : Social Science Research Council.
  • Arum, R. , & Roksa, J. (2011). Academically Adrift: Limited learning on college campuses . Chicago, IL : The University of Chicago Press.
  • Arum, R. , Roksa, J. , & Cho, E. (2011). Improving undergraduate learning: Findings and policy recommendations from the SSRC-CLA longitudinal project . New York, NY : Social Science Research Council.
  • Bailin, S. , & Siegel, H. (2003). Critical thinking. In N. Blake , P. Smeyers , R. Smith , & P. Standish (Eds.), The Blackwell guide to the philosophy of education (pp. 181–193). Oxford: Blackwell.
  • Barbaranelli, C. , Lee, C.S. , Vellone, E. , & Riegel, B. (2015). The problem with Cronbach’s Alpha: comment on Sijtsma and van der Ark (2015). Nursing Research , 64 (2), 140–145. doi:10.1097/NNR.0000000000000079
  • Bassok, M. , & Novick, L.R. (2012). Problem solving. In K.J. Holyoak & R.G. Morrison (Eds.), The Oxford handbook of thinking and reasoning (pp. 1–22). Oxford: Oxford University Press. doi: 10.1093/oxfordhb/9780199734689.013.0021
  • Bates, D.M. (2007, November 10). Re: [R] Confidence intervals for random effect BLUP’s . [Electronic mailing list message]. Retrieved from http://www.mail-archive.com/[email protected]/msg04913.html
  • Bates, D.M. (2010). lme4: Mixed-effects modeling with R . Retrieved from http://lme4.r-forge.r-project.org/lMMwR/lrgprt.pdf
  • Benjamin, R. , Klein, S.P. , Steedle, J.T. , Zahner, D. , Elliot, S. , & Patterson, J. (2013). The case for critical-thinking skills and performance assessment . New York, NY : Council for Aid to Education.
  • Berkow, S. , Virkstis, K. , Stewart, J. , Aronson, S. , & Donohue, M. (2011). Assessing individual frontline nurse critical thinking. The Journal of Nursing Administration , 41 , 168–171.
  • Black, B. (2012). An overview of a programme of research to support the assessment of critical thinking. Thinking Skills and Creativity , 7 (2), 122–133. doi:10.1016/j.tsc.2012.04.003
  • Bok, D.C. (2006). Our underachieving colleges: A candid look at how much students learn and why they should be learning more . Princeton, NJ: Princeton University Press.
  • Brudvig, T.J. , Dirkes, A. , Dutta, P. , & Rane, K. (2013). Critical thinking skills in health care professional students: A systematic review. Journal of Physical Therapy Education , 27 (3), 12–25.10.1097/00001416-201307000-00004
  • Brunt, B.A. (2005). Models, measurement, and strategies in developing critical-thinking skills. The Journal of Continuing Education in Nursing , 36 (6), 255–262.
  • Byrnes, J.P. , & Dunbar, K.N. (2014). The nature and development of critical-analytic thinking. Educational Psychology Review , 26 (4), 477–493. doi:10.1007/s10648-014-9284-0
  • Cacioppo, J.T. , Petty, R.E. , Feinstein, J.A. , & Jarvis, W.B.G. (1996). Dispositional differences in cognitive motivation: The life and times of individuals varying in need for cognition. Psychological Bulletin , 119 (2), 197–253. doi:10.1037/0033-2909.119.2.197
  • Cambridge Assessment Admissions Testing . (2017). Thinking skills assessment . Retrieved from http://www.admissionstesting.org/for-test-takers/thinking-skills-assessment/
  • Cambridge International Examinations . (2016). Frequently Asked Questions (FAQs): Cambridge International AS & A Level Thinking Skills (9694). Retrieved from http://www.cambridgeinternational.org/images/131019-frequently-asked-questions.pdf
  • Carter, A.G. , Creedy, D.K. , & Sidebotham, M. (2015). Evaluation of tools used to measure critical thinking development in nursing and midwifery undergraduate students: A systematic review. Nurse Education Today , 35 (7), 864–874. doi:10.1016/j.nedt.2015.02.023
  • Carter, A.G. , Creedy, D.K. , & Sidebotham, M. (2016). Efficacy of teaching methods used to develop critical thinking in nursing and midwifery undergraduate students: A systematic review of the literature. Nurse Education Today , 40 (7), 209–218. doi:10.1016/j.nedt.2016.03.010
  • Council for Aid to Education . (2014a). CLA+: Technical FAQ . New York, NY : Author.
  • Council for Aid to Education . (2014b). The case for performance-based assessment of critical-thinking skills . Retrieved from http://cae.org/flagship-assessments-cla-cwra/flagship-assessments-cla-cwra/resources/
  • Council for Aid to Education . (2015a). About CAE: History [Web page]. Retrieved from http://cae.org/about/history/
  • Council for Aid to Education . (2015b). Spring 2015 CLA+ mastery results: Sample university (Institutional report) . New York, NY : Author.
  • Council for Aid to Education . (2016). CLA+: National results, 201516 . Retrieved from http://cae.org/flagship-assessments-cla-cwra/cla/resources-for-cla/
  • Council for Aid to Education . (2017). Why CLA+ and CWRA+ [Web page]. Retrieved from http://cae.org/flagship-assessments-cla-cwra/why-cla-and-cwra/
  • Cedefop . (2013). Piloting a European employer survey on skill needs: Illustrative findings ( Research paper No. 36). Luxembourg: Publications Office of the European Union. doi:10.2801/3701
  • Clark, L.A. , & Watson, D. (1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment , 7 (3), 309–319. doi:10.1037/1040-3590.7.3.309
  • College Board . (2017). SAT suite of assessments [Web page]. Retrieved from https://collegereadiness.collegeboard.org/sat
  • Cortina, J.M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology , 78 (1), 98–104. doi:10.1037/0021-9010.78.1.98
  • Coutinho, S.A. (2006). The relationship between the need for cognition, metacognition, and intellectual task performance. Educational Research and Reviews , 1 (5), 162–164.
  • Cronbach, L.J. , & Meehl, P.E. (1955). Construct validity in psychological tests. Psychological Bulletin , 52 (4), 281–302. doi:10.1037/h0040957
  • Dalkey, N. , & Helmer, O. (1962). An experimental application of the Delphi method to the use of experts (Memorandum RM-727/1-abridged) . Santa Monica, CA: RAND Corporation.
  • Del Bueno, D.J. (1990). Experience, education, and nurses’ ability to make clinical judgments. Nursing and Health Care , 11 , 290–294.
  • Elder, L. (2007). Another brief conceptualization of critical thinking [Web page]. Retrieved from http://www.criticalthinking.org/pages/defining-critical-thinking/766
  • Ennis, R.H. (1962). A concept of critical thinking. Harvard Educational Review , 32 (1), 81–111.
  • Ennis, R.H. , Millman, J. , & Tomko, T.N. (1985). Cornell critical thinking tests level X and level Z manual . (3rd ed.). Pacific Grove, CA: Midwest Publications.
  • Ennis, R.H. (1987). A taxonomy of critical thinking dispositions and abilities. In J. Baron & R. Sternberg (Eds.), Teaching thinking skills theory and practice (pp. 9–26). New York, NY : W. H. Freeman.
  • Ennis, R.H. , & Weir, E. (1985). The Ennis-Weir critical thinking essay test . Pacific Grove, CA: Midwest Publications.
  • Erwin, T.D. , & Sebrell, K.W. (2003). Assement of critical thinking: ETS’s tasks in critical thinking. The Journal of General Education , 52 (1), 50–70.10.1353/jge.2003.0019
  • ETS . (2017). ETS proficiency profile [Web page]. Retrieved from https://www.ets.org/proficiencyprofile/about/
  • Evans, J.S.B.T. , & Stanovich, K.E. (2013). Dual-process theories of higher cognition: Advancing the debate. Perspectives on Psychological Science , 8 (3), 223–241. doi:10.1177/1745691612460685
  • Facione, P.A. (1990). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction – The Delphi Report (Executive Summary). Millbrae, CA: The California Academic Press.
  • Facione, P.A. (1992). The California Critical Thinking Skills Test manual . Millbrae, CA: California Academic Press.
  • Facione, P.A. , & Facione, N.C. (1992). The California Critical Thinking Dispositions Inventory test manual . Millbrae, CA: California Academic Press.
  • Frankham, J. (2016). Employability and higher education: The follies of the “productivity challenge” in the Teaching Excellence Framework. Journal of Education Policy , 1–14. doi:10.1080/02680939.2016.1268271
  • Furnham, A. , & Thorne, J.D. (2013). Need for cognition: Its dimensionality and personality and intelligence correlates. Journal of Individual Differences , 34 (4), 230–240. doi:10.1027/1614-0001/a000119
  • Glenn, D. (2011, January 18). New book lays failure to learn on colleges’ doorsteps. The Chronicle of Higher Education . Retrieved from http://www.chronicle.com/article/New-Book-Lays-Failure-to-Learn/125983/
  • Goldstein, H. (2011). Multilevel statistical models . (4th ed.). Chichester: Wiley.
  • Halpern, D.F. (2010). Halpern Critical Thinking Assessment manual. Vienna: Schuhfried GmbH.
  • Halpern, D.F. (2013). Critical thinking across the curriculum: A brief edition of thought and knowledge . New York, NY : Routledge.
  • Higher Education Funding Council for England . (2016). Learning and teaching excellence: Learning gain [Web page]. Retrieved from http://www.hefce.ac.uk/lt/lg/
  • Higher Education Funding Council for England . (2017). Qualifications obtained [Web page]. Retrieved from https://www.hesa.ac.uk/data-and-analysis/students/qualifications
  • Ibrahim, N.K. (2015). Critical literacy: Performance and reactions. Theory and Practice in Language Studies , 5 (4), 756–764. doi:10.17507/tpls.0504.11
  • Insight Assessment . (2013). California critical thinking disposition inventory (CCTDI) . San Jose, CA: The Californian Academic Press.
  • Johnson, R.H. , & Hamby, B. (2015). A meta-level approach to the problem of defining “critical thinking”. Argumentation , 29 (4), 417–430. doi:10.1007/s10503-015-9356-4
  • Kane, M.T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement , 50 (1), 1–73. doi:10.1111/jedm.12000
  • Klein, S.P. (1996). The costs and benefits of performance testing on the bar examination. The Bar Examiner , 65 (3), 13–20.
  • Klein, S.P. (2008). Characteristics of hand and machine-assigned scores to college students’ answers to open-ended tasks. In Probability and statistics: Essays in honor of David A. Freedman (Vol. 2, pp. 76–89). Beachwood, OH: Institute of Mathematical Statistics. doi:10.1214/193940307000000392 10.1214/193940307000000392
  • Klein, S.P. , Benjamin, R. , Shavelson, R.J. , & Bolus, R. (2007). The Collegiate Learning Assessment: Facts and fantasies. Evaluation Review , 31 (5), 415–439. doi:10.1177/0193841X07303318
  • Klein, S.P. , Kuh, G.D. , Chun, M. , Hamilton, L. , & Shavelson, R. (2005). An approach to measuring cognitive outcomes across higher education institutions. Research in Higher Education , 46 (3), 251–276. doi:10.1007/s11162-004-1640-3
  • Klein, S.P. , Liu, O.L. , & Sconing, J. (2009). Test Validity Study (TVS) report . Retrieved from http://cae.org/images/uploads/pdf/13_Test_Validity_Study_Report.pdf
  • Kong, L.-N. , Qin, B. , Zhou, Y. , Mou, S. , & Gao, H.-M. (2014). The effectiveness of problem-based learning on development of nursing students’ critical thinking: A systematic review and meta-analysis. International Journal of Nursing Studies , 51 (3), 458–469. doi:10.1016/j.ijnurstu.2013.06.009
  • Kuechler, W.L. , & Simkin, M.G. (2010). Why is performance on multiple-choice tests and constructed-response tests not more closely related? Theory and an empirical test. Decision Sciences Journal of Innovative Education , 8 (1), 55–73. doi:10.1111/j.1540-4609.2009.00243.x
  • Lai, E.R. (2011). Critical thinking: A literature review . Retrieved from http://images.pearsonassessments.com/images/tmrs/CriticalThinkingReviewFINAL.pdf
  • Lawson, T.J. , Jordan-Fleming, M.K. , & Bodle, J.H. (2015). Measuring psychological critical thinking. Teaching of Psychology , 42 (3), 248–253. doi:10.1177/0098628315587624
  • Lee, J. , Lee, Y. , Gong, S. , Bae, J. , & Choi, M. (2016). A meta-analysis of the effects of non-traditional teaching methods on the critical thinking abilities of nursing students. BMC Medical Education , 16 (1), 240. doi:10.1186/s12909-016-0761-7
  • Lindsay, T.K. (2013). The likelihood of higher-education reform. Society , 50 (3), 236–244. doi:10.1007/s12115-013-9649-x
  • Lipman, M. (1987). Critical thinking: What can it be? Analytic Teaching , 8 (1), 5–12.
  • Liu, O.L. , Frankel, L. , & Roohr, K.C. (2014). Assessing critical thinking in higher education: Current state and directions for next-generation assessment. ETS Research Report Series , 2014 (1), 1–23. doi:10.1002/ets2.12009
  • Liu, O.L. , Mao, L. , Frankel, L. , & Xu, J. (2016). Assessing critical thinking in higher education: The HEIghtenTM approach and preliminary validity evidence. Assessment & Evaluation in Higher Education , 41 (5), 677–694. doi:10.1080/02602938.2016.1168358
  • McGrath, C.H. , Guerin, B. , Harte, E. , Frearson, M. , & Manville, C. (2015). Learning gain in higher education . Santa Monica, CA: RAND Corporation.
  • McKelvie, S.J. (1992). Does memory contaminate test – Retest reliability? Journal of General Psychology , 119 (1), 59–72.10.1080/00221309.1992.9921158
  • McPeck, J.E. (1981). Critical thinking and education . New York, NY : St. Martin’s Press.
  • Menchaca, F. (2014). Start a new fire: Measuring the value of academic libraries in undergraduate learning. Portal: Libraries and the Academy , 14 (3), 353–367. doi:10.1353/pla.2014.0020
  • Messick, S. (1984). The psychology of educational measurement. Journal of Educational Measurement , 21 (3), 215–237.10.1111/jedm.1984.21.issue-3
  • Messick, S. (1989). Validity. In: R.L. Linn (Ed.), Educational measurement (pp. 13–103). New York, NY : Macmillan.
  • Murphy, P.K. , Rowe, M.L. , Ramani, G. , & Silverman, R. (2014). Promoting critical-analytic thinking in children and adolescents at home and in school. Educational Psychology Review , 26 (4), 561–578. doi:10.1007/s10648-014-9281-3
  • Newton, P.E. (2007). Clarifying the purposes of educational assessment. Assessment in Education , 14 (2), 149–170. doi:10.1080/09695940701478321
  • Newton, P.E. , & Shaw, S.D. (2015). Disagreement over the best way to use the word “validity” and options for reaching consensus. Assessment in Education: Principles, Policy & Practice , 23(2), 178–197. doi:10.1080/0969594X.2015.1037241
  • Niu, L. , Behar-Horenstein, L.S. , & Garvan, C.W. (2013). Do instructional interventions influence college students’ critical thinking skills? A meta-analysis. Educational Research Review , 9 , 114–128. doi:10.1016/j.edurev.2012.12.002
  • O’Hare, L. , & McGuinness, C. (2015). The validity of critical thinking tests for predicting degree performance: A longitudinal study. International Journal of Educational Research , 72 , 162–172. doi:10.1016/j.ijer.2015.06.004
  • Opposs, D. , & He, Q. (2011). The reliability programme: Final report of the technical advisory group ( No. 4828). Coventry: Office of Qualifications and Examinations Regulation.
  • Pascarella, E.T. , & Terenzini, P.T. (2005). How college affects students: a third decade of research . San Francisco, CA: John Wiley and Sons.
  • Paul, R. (1992). Critical thinking: What, why, and how. New Directions for Community Colleges , 1992 (77), 3–24. doi:10.1002/cc.36819927703
  • Popham, W.J. (1997). What’s wrong – And what’s right – With rubrics. Educational Leadership , 55 (2), 72–75.
  • Pucer, P. , Trobec, I. , & Žvanut, B. (2014). An information communication technology based approach for the acquisition of critical thinking skills. Nurse Education Today , 34 (6), 964–970.
  • Romeo, E.M. (2010). Quantitative research on critical thinking and predicting nursing students’ NCLEX-RN performance. Journal of Nursing Education , 49 (7), 378–386. doi:10.3928/01484834-20100331-05
  • Roy, T. (2017). Symbolic logic: A accessible introduction to serious mathematical logic. ( Version 7.3). Retrieved from http://rocket.csusb.edu/~troy/SLmain.pdf
  • Sadler, D.R. (2009). Indeterminacy in the use of preset criteria for assessment and grading. Assessment & Evaluation in Higher Education , 34 (2), 159–179. doi:10.1080/02602930801956059
  • Samson, P.L. (2016). Critical thinking in social work education: A research synthesis. Journal of Social Work Education , 52 (2), 147–156. doi:10.1080/10437797.2016.1151270
  • Scriven, M. , & Paul, R. (1987). Critical thinking. 8th Annual International Conference on Critical Thinking and Education Reform [Web page]. Retrieved from http://www.criticalthinking.org/pages/defining-critical-thinking/766
  • Shavelson, R.J. , & Huang, L. (2003). Responding responsibly to the frenzy to assess learning in higher education. Change , 1383 (January/February), 11–19. doi:10.1080/00091380309604739
  • Shaw, S.D. , & Crisp, V. (2012). An approach to validation: Developing and applying an approach for the validation of general qualifications. Research Matters: A Cambridge Assessment Publication , Special Issue 3. Retrieved from http://www.cambridgeassessment.org.uk/our-research/all-published-resources/research-matters/
  • Siegel, H. (1988). Educating reason: Rationality, critical thinking and education . New York, NY : Routledge.
  • Sireci, S.G. (1998). The construct of content validity. Social Indicators Research , 45 (1), 83–117. doi:10.1023/A:1006985528729
  • Sosu, E.M. (2013). The development and psychometric validation of a Critical Thinking Disposition Scale. Thinking Skills and Creativity , 9 , 107–119. doi:10.1016/j.tsc.2012.09.002
  • Stanovich, K.E. (2016). The comprehensive assessment of rational thinking. Educational Psychologist , 51 (1), 23–34. doi:10.1080/00461520.2015.1125787
  • Stedman, N.L.P. , Irani, T.A. , Friedel, C. , Rhoades, E.B. , & Ricketts, J.C. (2009). Relationships between critical thinking disposition and need for cognition among undergraduate students enrolled in leadership courses. NACTA Journal , 53 (3), 62–70.
  • Steedle, J.T. (2012). Selecting value-added models for postsecondary institutional assessment. Assessment & Evaluation in Higher Education , 37 (6), 637–652. doi:10.1080/02602938.2011.560720
  • Steedle, J.T. (2014). Motivation filtering on a multi-institution assessment of general college outcomes. Applied Measurement in Education , 27 (1), 58–76. doi:10.1080/08957347.2013.853072
  • Stupple, E.J.N. , Maratos, F.A. , Elander, J. , Hunt, T.E. , Cheung, K.Y.F. , & Aubeeluck, A.V. (2017). Development of the Critical Thinking Toolkit (CriTT): A measure of student attitudes and beliefs about critical thinking. Thinking Skills and Creativity , 23 , 91–100. doi:10.1016/j.tsc.2016.11.007
  • Tavakol, M. , & Dennick, R. (2011). Making sense of Cronbach’s alpha. International Journal of Medical Education , 2 , 53–55. doi:10.5116/ijme.4dfb.8dfd
  • Thayer-Bacon, B. (1998). Transforming and redescribing critical thinking: Constructive thinking. Studies in Philosophy and Education , 17 (2/3), 123–148. doi:10.1023/A:1005166416808
  • Walker, P. , & Finney, N. (1999). Skill development and critical thinking in higher education. Teaching in Higher Education , 4 (4), 531–547. doi:10.1080/1356251990040409
  • Watson, G. , & Glaser, E.M. (1980). Watson-Glaser critical thinking appraisal manual . Cleveland, OH: Psychological Corp.
  • Winterbotham, M. , Vivian, D. , Shury, J. , Davies, B. , & Kik, G. (2014). UK commission’s employer skills survey 2013: UK results ( Evidence report No. 81). Retrieved from https://www.gov.uk/government/publications/ukces-employer-skills-survey-2013
  • Wolf, R. , Zahner, D. , & Benjamin, R. (2015). Methodological challenges in international comparative post-secondary assessment programs: Lessons learned and the road ahead. Studies in Higher Education , 40 (3), 471–481. doi:10.1080/03075079.2015.1004239
  • Zahner, D. (2014a). Reliability and validity of CLA+ . New York, NY : Council for Aid to Education.
  • Zahner, D. (2014b). Standard-setting study final report . New York, NY : Council for Aid to Education.
  • Zahner, D. , & James, J.K. (2015). Predictive validity of a critical thinking assessment for post-college outcomes . New York, NY : Council for Aid to Education.
  • Zahner, D. , & Steedle, J.T. (2015). Comparing longitudinal and cross-sectional school effect estimates in postsecondary education . Paper presented at the 2015 National Council on Measurement in Education Annual Meeting, Chicago, IL.
  • Zuriguel Pérez, E. , Lluch Canut, M.T. , Falcó Pegueroles, A. , Puig Llobet, M. , Moreno Arroyo, C. , & Roldán Merino, J. (2015). Critical thinking in nursing: Scoping review of the literature. International Journal of Nursing Practice , 21 (6), 820–830. doi:10.1111/ijn.12347