The Multimodal Listening Test in a High-Stakes Context: Gender-Neutral or not?

ABSTRACT

In this study, we used the Rasch measurement to investigate the fairness of the listening section of a national computerized high-stakes English test for differential item functioning (DIF) across gender subgroups. The computerized test format inspired us to investigate whether the items measure listening comprehension differently for females and males. Exploring the functioning of novel task types including multimodal materials such as videos and pictures was especially interesting. Firstly, the unidimensionality and local independence of the data were examined as preconditions for DIF analysis. Secondly, the authors explored the performance of female and male students through DIF analysis using the Rasch measurement. The uniform DIF analysis showed that 25 items (out of 30 items) displayed DIF and favored different gender subgroups, whereas the effect size was not meaningful. The non-uniform DIF analysis revealed several items exhibiting DIF with a moderate to large effect size, favoring various gender and ability groups. Explanations for DIF are hypothesized. Finally, implications of the study regarding test development and fairness are discussed.

Acknowledgments

We express our gratitude to the reviewers of The International Journal of Listening for their insightful comments. We would like to thank Dr. Vahid Aryadoust for inspiring us and commenting on this article.

Disclosure statement

Dr. Anna von Zansen worked for the Matriculation Examination Board during the computerization phase of the examination 2013-2016.

Dr. Raili Hilden works as the chair of the Finnish Matriculation Examination Language Section 2016-2021.