On the Complexity of Item Response Theory Models

ABSTRACT

Complexity in item response theory (IRT) has traditionally been quantified by simply counting the number of freely estimated parameters in the model. However, complexity is also contingent upon the functional form of the model. We examined four popular IRT models—exploratory factor analytic, bifactor, DINA, and DINO—with different functional forms but the same number of free parameters. In comparison, a simpler (unidimensional 3PL) model was specified such that it had 1 more parameter than the previous models. All models were then evaluated according to the minimum description length principle. Specifically, each model was fit to 1,000 data sets that were randomly and uniformly sampled from the complete data space and then assessed using global and item-level fit and diagnostic measures. The findings revealed that the factor analytic and bifactor models possess a strong tendency to fit any possible data. The unidimensional 3PL model displayed minimal fitting propensity, despite the fact that it included an additional free parameter. The DINA and DINO models did not demonstrate a proclivity to fit any possible data, but they did fit well to distinct data patterns. Applied researchers and psychometricians should therefore consider functional form—and not goodness-of-fit alone—when selecting an IRT model.

KEYWORDS:

• Bifactor model
• diagnostic classification model
• item response theory
• minimum description length
• model evaluation

Article information

Conflict of Interest Disclosures: Each author signed a form for disclosure of potential conflicts of interest. No authors reported any financial or other conflicts of interest in relation to the work described.

Ethical Principles: The authors affirm having followed professional ethical guidelines in preparing this work. These guidelines include obtaining informed consent from human participants, maintaining ethical treatment and respect for the rights of human or animal participants, and ensuring the privacy of participants and their data, such as ensuring that individual participants cannot be identified in reported results or from publicly available original or archival data.

Funding: This work was partially supported by Grant R305D140046 from the Institute of Education Sciences.

Role of the Funders/Sponsors: None of the funders or sponsors of this research had any role in the design and conduct of the study; collection, management, analysis, and interpretation of data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

Acknowledgments: The ideas and opinions expressed herein are those of the authors alone, and endorsement by the authors' institutions or the Institute of Education Sciences is not intended and should not be inferred.

Notes

1 A bit is the base-2 unit of information (i.e., 0 or 1) (see e.g., Shannon & Weaver, 1949); a nat is the base-e unit of information (Boulton & Wallace, 1970). One nat ≈ 1.443 bits.

2 An accessible introductory overview of MDL is given by Grünwald (2005), and explicit comparisons with frequentist and Bayesian methods are given by Vitányi and Li (2000), Markon and Kreuger (2004), and Lee and Pope (2006).

3 We also explored FP by investigating the S-X² item-fit index (Orlando & Thissen, 2000; 2003), the D² latent distribution fit index (Li & Cai, 2012), and the marginal χ² values of each of the five IRT models. Due to page limitations, discussion of these indices and their bearing on FP is included in the online supplementary material.