Exploring task features that predict psychometric quality of test items: the case for the Dutch driving theory exam

Abstract

This study reports on an Evidence Centered Design (ECD) project in the Netherlands, involving the theory exam for prospective car drivers. In particular, we illustrate how cognitive load theory, task-analysis, response process models, and explanatory item-response theory can be used to systematically develop and refine task models. Based on a cognitive model for driving, 353 existing items involving rules of priority at intersections, were coded on intrinsic task features and task presentation features. Hierarchical regression analyses were carried out to determine the contribution of task features to item difficulty and item discrimination. A substantial proportion of variance in both item difficulty and item discrimination parameters could be explained by intrinsic task-features, including rules and signs (25%, 18.6%), task-intersection features (13.4%, 14.1%), and a smaller small proportion to item presentation features (3.5%, 7.1%) of the total variance. It is concluded that the systematic approach of discerning task features and determining the impact on item parameters has added value as an ECD-tool for evaluating existing assessments that are planned to be innovated. The paper concludes with a discussion of practical implications.

Keywords:

• Cognitive load theory
• evidence centered design
• explanatory item response theory
• item accessibility
• task features

Acknowledgments

We would like to express our gratitude to the Theory tests division of CBR, The Dutch agency responsible for driving exams in the Netherlands, for sharing the anonymized test data for this study. CBR had no active role in this study. The authors are fully responsible for the study and the conclusions drawn from it.

Notes

1 The presented item in Figure 1 is only meant for illustrating features of an intersection task, but it is not a real item used in the study. Disclosure of typical items in the Dutch Theory exam, employing high-end simulated pictures, is prohibited for reasons of test security.