Predicting the difficulty of complex logical reasoning problems

Abstract

The aim of the present research was to develop a difficulty model for logical reasoning problems involving complex ordered arrays used in the Graduate Record Examination. The approach used involved breaking down the problems into their basic cognitive elements such as the complexity of the rules used, the number of mental models required to represent the problem, and question type. Weightings for these different elements were derived from two experimental studies and from the reasoning literature. Based on these weights, difficulty models were developed which were then tested against new data. The models had excellent predictive validity and showed the relative influence of rule based factors and factors relating to the number of underlying models. Different difficulty models were needed for different question types, suggesting that people used a variety of approaches and, at a wider level, that both mental models and mental rules may be used in reasoning.