Detection and Validation of Unscalable Item Score Patterns Using Item Response Theory: An Illustration with Harter's Self-Perception Profile for Children

Abstract

We illustrate the usefulness of person-fit methodology for personality assessment. For this purpose, we use person-fit methods from item response theory. First, we give a nontechnical introduction to existing person-fit statistics. Second, we analyze data from Harter's (1985) Self-Perception Profile for Children (Harter, 1985) in a sample of children ranging from 8 to 12 years of age (N = 611) and argue that for some children, the scale scores should be interpreted with care and caution. Combined information from person-fit indexes and from observation, interviews, and self-concept theory showed that similar score profiles may have a different interpretation. For some children in the sample, item scores did not adequately reflect their trait level. Based on teacher interviews, this was found to be due most likely to a less developed self-concept and/or problems understanding the meaning of the questions. We recommend investigating the scalability of score patterns when using self-report inventories to help the researcher interpret respondents' behavior correctly.

Notes

¹An alternative would have been to use a multidimensional IRT model and a multidimensional person-fit statistic. We did not use a multidimensional IRT model because there is not much experience with the application of multidimensional IRT models to personality data. Another reason is the relative complexity to understand the outcomes of these models for the school teachers that were involved in this project. Therefore, we used a relatively simple IRT model to get a first impression about data quality and quality of individual item score patterns. We do not expect that results would have been dramatically different had we used more complex models, but future research may use these models to answer that question.

²Group differences in scalability may be the result of differential item functioning (DIF). We investigated DIF by means of MSP5.0 for young and old children and girls and boys. There were only small differences in the item mean scores for the different groups. Furthermore, several studies (e.g., Meijer & Van Krimpen-Stoop, 2001) have shown that there was no effect of DIF on the classification of an item score patterns as fitting or misfitting; that is, person-fit results were similar.

^a n = 119.

^b n = 140.

^c n = 147.

^d n = 194.

³We used the program MSP5.0. Further information about this program can be obtained from K. Sijtsma ([email protected]) or W. H. M. Emons ([email protected]). See also http://www.scienceplus.nl/scienceplus/main/softwareshop/msp.jsp.

* = frequency count on a conformal item-score pair. From Introduction to Nonparametric Item Response Theory (p. 134), by K. Sijtsma and I. W. Molenaar, 2002, Thousand Oaks, CA: Sage.