A Comparison of Limited-Information Test Statistics for a Response Style MIRT Model

Abstract

An increased use of models for measuring response styles is apparent in recent years with the multidimensional nominal response model (MNRM) as one prominent example. Inclusion of latent constructs representing extreme (ERS) or midpoint response style (MRS) often improves model fit according to information criteria. However, a test of absolute model fit is often not reported even though it could comprise an important piece of validity evidence. Limited information test statistics are candidates for this task, including the full ($M_2$), ordinal ($M_2^{\mathrm{*}}$), and mixed ($C_2$) statistics, which differ in whether additional collapsing of univariate or bivariate contingency tables is conducted. Such collapsing makes sense when item categories are ordinal, which may not hold under the MNRM. More generally, limited information test statistics have gone unevaluated under nominal data and non-ordinal latent trait models. We present a simulation study evaluating the performance of $M_2,$ $M_2^{\mathrm{*}},$ and $C_2$ with the MNRM. Manipulated conditions included sample size, presence and type of response style, and strength of item slopes on substantive and style dimensions. We found that $M_2$ sometimes had inflated Type I error rates, $M_2^{\mathrm{*}}$ always had little power, and $C_2\mathrm{}$ lacked power under some conditions. $M_2$ and $C_2$ may provide complementary and valuable information regarding model fit.

Keywords:

• Model fit
• limited-information test statistics
• Likert-type items
• multidimensional item response theory
• nominal response model
• response styles

Article information

Conflict of interest disclosures: Each author signed a form for disclosure of potential conflicts of interest. No authors reported any financial or other conflicts of interest in relation to the work described.

Ethical principles: The authors affirm having followed professional ethical guidelines in preparing this work. These guidelines include obtaining informed consent from human participants, maintaining ethical treatment and respect for the rights of human or animal participants, and ensuring the privacy of participants and their data, such as ensuring that individual participants cannot be identified in reported results or from publicly available original or archival data.

Funding: This work was supported by Grant RGPIN-2018-05357 and DGECR-2018-00083 from the National Science and Engineering Research Council of Canada (NSERC).

Role of the funders/sponsors: None of the funders or sponsors of this research had any role in the design and conduct of the study; collection, management, analysis, and interpretation of data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

Acknowledgements: The authors would like thank Li Cai for their comments on prior versions of this manuscript. The ideas and opinions expressed herein are those of the authors alone, and endorsement by the authors’ institutions is not intended and should not be inferred.

Correction Statement

This article has been republished with minor changes. These changes do not impact the academic content of the article.

Notes

1 In addition to sparsity, all such statistics may require numerical integration, which is feasible for models with few latent traits, but still problematic for models with many latent traits.

2 We have retained all participants regardless of their standing on quality control checks. Our use of the subsequent models is mainly for illustrative purposes, and we do not make any firm substantive conclusions from the results of these analyses.

3 If items differ greatly in overall difficulty, these high frequencies may not cluster along the diagonal, but will still have a similar pattern where category pairs further toward at least one of the off-diagonals will have the lowest frequencies.

4 To establish the asymptotic distribution for the family, it is necessary that: 1) ${\mathbf{L}}_{\mathit{\eta }}$ has full row rank, $s,$ and $1_{\mathit{C}}^{\prime }$ is not in its row span; and 2) ${\mathbf{\Delta }}_{\mathit{\eta }}$ has full column rank, $v$ (Joe & Maydeu-Olivares, 2010, p. 396).

5 Despite what appear to be slightly small standardized loadings for these items (e.g., Table 5; computed using flexMIRT^®), the strong and weak conditions for the unidimensional model and a standard normal latent trait correspond to marginal reliabilities of .91 and .84, respectively. The information provided for style factors is expected to be less since the effective number of categories per item is fewer (e.g., see also Falk & Ju, 2020).

6 In support of this claim, we computed model-implied probabilities for all $5^{10}$ response patterns for a 10-item (5 categories per item), two-factor (substantive + ERS) model with a .3 correlation among latent dimensions. As an example, fitting alternative misspecified models (a single substantive dimension) to these model-implied proportions typically yielded near zero test statistics for $M_2^{*},$ even if a wildly inflated sample size was specified for the fitted model (e.g., close to 1 million). This strategy was deemed not appropriate for the 12-item measures in our empirical example and simulation study, as doing computations on $5^{12}$ patterns was too RAM and processor intensive.