Liberal and Conservative Differential Item Functioning Detection Using Mantel-Haenszel and SIBTEST: Implications for Type I and Type II Error Rates

Abstract

The aim of this work was to determine, in terms of Type I and Type II error rates, the risks of applying various statistical procedures for evaluating differential item functioning. To this end, the authors carried out a simulation study in which the Mantel-Haenszel and SIBTEST procedures were applied in conjunction. The variables manipulated were sample size and distribution of ability between groups. Results indicated that, although there was a high rate of agreement between the procedures, the joint Type I and Type II error rate may vary substantially from that obtained when each of the procedures was applied separately. Furthermore, the authors analyzed empirical data to obtain information complementary to the Monte Carlo study. Two strategies that minimize each type of error and some issues of a practical nature are discussed.

Keywords:

• differential item functioning
• empirical study
• Mantel-Haenszel procedure
• SIBTEST
• simulation study
• Type I error rate
• Type II error rate