How does the number of actions on constructed-response items relate to test-taking effort and performance?

ABSTRACT

Collecting process data in computer-based assessments provides opportunities to describe examinee behaviour during a test-taking session. The number of actions taken by students while interacting with an item is in this context a variable that has been gaining attention. The present study aims to investigate how the number of actions performed on constructed-response items relates to self-reported effort, performance, and item cluster position in the test. The theory of planned behaviour was used as an interpretative framework. Data from two item clusters of the 2015 Swedish Programme for International Student Assessment (PISA) Science administration were utilised. Results showed that the number of actions was significantly related to performance on the items, self-reported test-taking effort, and cluster position. Latent variable models were examined separately for performance-level groups. Overall, the number of actions performed on constructed-response items as a behavioural indicator in testing situations may be useful in gauging test-taking engagement.

KEYWORDS:

• Constructed-response items
• number of actions
• test-taking effort
• engagement
• cluster position
• item scores

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1 One item in Cluster 11 could take a value = 0 for incorrect, a value = 1 for partially correct, and a value = 2 for fully correct response. One item in Cluster 10 took a value = 01 for incorrect, a value = 11 or 12 for partially correct, and a value = 21 for fully correct response. The particular item in Cluster 10 was recoded in a way that students got a value = 0 for incorrect, a value = 1 for partially correct, and a value = 2 for fully correct response.

2 Initially, a logarithmic transformation was attempted, but the transformed number of actions variable did not approach normality. For this reason, the square root transformation of number of actions was preferred.

3 Invariance analysis by science performance groups was carried out for each cluster separately. Configural invariance was supported, but metric or partial metric invariance was not. Therefore, it was not possible to compare the magnitude of path coefficients across groups with a formal test.

Additional information

Notes on contributors

Militsa Ivanova

Militsa Ivanova is a school psychologist, currently studying for a PhD in Clinical Psychology in the Department of Psychology at the University of Cyprus. The focus of her research involves using process data to explore examinees’ test-taking behaviour in international large-scale assessments. Her research interests include investigating the effect of achievement motivation, test-taking effort, and engagement on test performance and on the psychometric properties of a test.

Michalis Michaelides

Michalis Michaelides is an assistant professor in Research Methodology in the Department of Psychology at the University of Cyprus. His research interests include psychometric and methodological issues in psychological and educational assessment. Using latent variable modelling, he has examined method effects due to wording in self-report scales, and validation issues in the adaptation of psychometric instruments. More recently, he has been exploring data in international, large-scale studies of educational assessment, such as TIMSS and PISA, to understand the role of effort and motivation in achievement.

Hanna Eklöf

Hanna Eklöf is an associate professor at the Department of Applied Educational Science, Umeå University, Sweden. She has a PhD in educational measurement from Umeå University. Her research interests are in the field of educational psychology and, more specifically, the psychology of test-taking, test-taking behaviour and motivation, large-scale testing, and assessment quality issues. She is currently the principal investigator of a project funded by the Swedish Research Council, where student effort and test-taking behaviour is investigated through self-report and process data in PISA. She is also co-investigator in projects investigating math anxiety, working memory, and performance.