Threats to the validity of the Collegiate Learning Assessment (CLA+) as a measure of critical thinking skills and implications for Learning Gain

Abstract

The University of Reading Learning Gain project is a three-year longitudinal project to test and evaluate a range of available methodologies and to draw conclusions on what might be the right combination of instruments for the measurement of Learning Gain in higher education. This paper analyses the validity of a measure of critical thinking skills, the Collegiate Learning Assessment (CLA+) and the implications of using this standardised test as a proxy for Learning Gain. The paper reviews five inferences regarding the interpretations and use of test scores: construct representation, scoring, generalisation, extrapolation and decision-making. Each section reviews some of the available evidence in support of the claims the CLA+ makes and the threats to their validity. The possible impact of these issues on Learning Gain in the UK is considered.

Keywords:

• Learning gain
• collegiate learning assessment
• critical thinking
• validity
• reliability

Notes

1. Broadly speaking, an argument is valid if the internal logic is consistent and the conclusion follows from its premises, but it is sound if it is valid and the premises are true (Roy, 2017).

2. It is unclear why Zahner and Steedle (2015) decided to use two different model equations. The authors seem convinced that the two models are substantively different; so much that they call the first ‘CLA value-added’ model and the second a ‘random effects’ model. In fact, both are random effects models. The level-2 residuals are shrunken in both cases, whereas Zahner and Steedle (2015) seem to suggest that they are not. The two models differ only in the choice of covariates; when the authors switch datasets, the only difference between the models is that one does not include the aggregate SATs scores. It is therefore unsurprising to observe high correlations between predictions.

3. This relationship would not be unique to the CLA+. For example, the California Critical Thinking Skills Test was found to be correlated to university marks (O’Hare & McGuinness, 2015).

4. Notice that even logicists do not consider CT only as a matter for logic. The ethics of critical thinking were discussed in Facione (1990), as was its being a ‘liberating’ force (p. 2). There is wide agreement that being rational ‘also requires an open-minded yet critical approach to one’s own thinking as well as that of others’ (Black, 2012, p. 125), and critical thinking skills have been viewed as playing an important role in ‘solid liberal education’ (Facione, 1990, p. 5). Bailin and Siegel (2003) suggested that ‘having the ability to think critically requires […] having the ability to ascertain the goodness of candidate reasons [for or against a judgement]’ (p. 182), but they also acknowledged that the criteria whereby a reason is to be considered good need not draw exclusively from the sphere of logic.