Exploring differential bundle functioning in mathematics by gender: the effect of hierarchical modelling

Abstract

Researchers interested in exploring substantive group differences are increasingly attending to bundles of items (or testlets): the aim is to understand how gender differences, for instance, are explained by differential performances on different types or bundles of items, hence differential bundle functioning (DBF). Some previous work has modelled hierarchies in data in this context or considered item responses within persons, but here we model the bundles themselves as explanatory variables at the item level potentially explaining significant intra-class correlation due to gender differences in item difficulty, and thus explaining variation at the second item level. In this study, we analyse DBF using single- and two-level models (the latter modelling random item effects, which models responses at Level 1 and items at Level 2) in a high-stakes National Mathematics test. The models show comparable regression coefficients but the statistical significances of the two-level models are smaller due to the larger values of the estimated standard errors. We discuss the contrasting relevance of this effect for test developers and gender researchers.

Keywords:

• logistic regression
• hierarchical modelling
• differential bundle functioning
• gender differences
• validity

Acknowledgements

The authors thank Malcolm Hayes and the Qualification Curriculum Authority of England for granting access to the data set used in this study. The authors also thank the anonymous reviewers for their constructive comments and suggestions.