A signal detection–item response theory model for evaluating neuropsychological measures

ABSTRACT

Introduction: Models from signal detection theory are commonly used to score neuropsychological test data, especially tests of recognition memory. Here we show that certain item response theory models can be formulated as signal detection theory models, thus linking two complementary but distinct methodologies. We then use the approach to evaluate the validity (construct representation) of commonly used research measures, demonstrate the impact of conditional error on neuropsychological outcomes, and evaluate measurement bias.

Method: Signal detection–item response theory (SD–IRT) models were fitted to recognition memory data for words, faces, and objects. The sample consisted of U.S. Infantry Marines and Navy Corpsmen participating in the Marine Resiliency Study. Data comprised item responses to the Penn Face Memory Test (PFMT; N = 1,338), Penn Word Memory Test (PWMT; N = 1,331), and Visual Object Learning Test (VOLT; N = 1,249), and self-report of past head injury with loss of consciousness.

Results: SD–IRT models adequately fitted recognition memory item data across all modalities. Error varied systematically with ability estimates, and distributions of residuals from the regression of memory discrimination onto self-report of past head injury were positively skewed towards regions of larger measurement error. Analyses of differential item functioning revealed little evidence of systematic bias by level of education.

Conclusions: SD–IRT models benefit from the measurement rigor of item response theory—which permits the modeling of item difficulty and examinee ability—and from signal detection theory—which provides an interpretive framework encompassing the experimentally validated constructs of memory discrimination and response bias. We used this approach to validate the construct representation of commonly used research measures and to demonstrate how nonoptimized item parameters can lead to erroneous conclusions when interpreting neuropsychological test data. Future work might include the development of computerized adaptive tests and integration with mixture and random-effects models.

KEYWORDS:

• Assessment
• item response theory
• neuropsychology
• recognition memory
• signal detection theory
• traumatic brain injury

Acknowledgments

We would like to acknowledge additional contributions from the MRS administrative core (Anjana Patel, Andrew De La Rosa, Elin Olsson) as well as the numerous clinician-interviewers and data collection staff who contributed to the project. We would like to thank Allison Port for preparing the neurocognitive data. Finally, we also wish to thank Marine and Navy Personnel who participated in the study.

Disclosure statement

No potential conflict of interest was reported by the authors.

Supplemental material

Supplemental data for this article can be accessed here.

Notes

1. In previous work, we have found that the additional parameter of the unequal variance SDT model does not meaningfully contribute to the measurement of individual differences. Moreover, this model is not commonly used in applied neuropsychological work. Nonetheless, future studies may wish to explore the impact of the equal variance assumption on the estimation and interpretation of model parameters.

2. With respect to the choice of link function, f, the probit has been more strongly associated with SDT, and the logit has been more strongly associated with Luce’s choice model (Luce, 1959, 1963); however, as with IRT, several authors have noted that under the right parameterization the results are nearly equivalent so long as the appropriate scaling constant is used (e.g., DeCarlo, 1998; Kornbrot, 2006; Snodgrass & Corwin, 1988; Wickens, 2002).

3. For multidimensional models, SE_θ is defined with respect to a likelihood surface. The direction of descent along the surface impacts the value of SE_θ. Here, we define SE_θ with respect to the steepest descent along a line from the origin of the space (see Reckase, 2009).

4. Choosing identification constraints—as is required for latent variable models—requires special care. It is interpretively convenient, though not necessary, to scale estimates in a manner that is consistent with values that are obtained using closed-form SDT solutions (see Snodgrass & Corwin, 1988). For this purpose, identification can be achieved by (a) constraining the α parameters according to Equation (12), (b) freely estimating the θ_d parameters, (c) freely estimating the θ_Ccenter in the τ constrained model but constraining their mean to 0 in the τ unconstrained model, and (d) constraining all τ parameters to 0 in the τ constrained model but constraining the mean of the τ parameters to 0 in the τ unconstrained model. At the time of writing, the mirt package did not allow the mean of the τ parameters to be constrained; thus, we instead fixed the θ_d′ mean to 0 during estimation and then rescaled parameters after estimation to achieve the desired scaling in the τ unconstrained model. Also, we rescaled estimates of θ_d′ and θ_Ccenter to be consistent with the normal metric. Example R code that simulates data and then estimates both SD–IRT models is provided in online supplemental material.

Additional information

Funding

This work was supported, in part, by the National Institute of Mental Health (NIMH) [grant number MH089983], [grant number MH019112], [grant number MH096891], and [grant number MH102420]; the Dowshen Program for Neuroscience; and the Navy Bureau of Medicine and Surgery [grant number N62645-11-C-4037]