Using Reliability Generalization Methods to Explore Measurement Error: An Illustration Using the MMPI–2 PSY–5 Scales

Abstract

Reliability generalization (RG) is a meta-analytic technique that allows for the systematic examination of variation in score reliability for different samples of test takers; this procedure is based on the recognition that reliability is not a stable property of a test but is sample dependent. As a demonstration of an RG analysis, I obtained 63 reliability coefficients for each of the MMPI–2 (Butcher et al., 2001) Personality Psychopathology 5 (Harkness, McNulty, & Ben-Porath, 1995) scales. The overall variability of alpha coefficients supports the argument that reliability is sample dependent and underscores the need for researchers to calculate reliability estimates based on their research samples rather than simply citing published alpha coefficients as evidence of score reliability. I observed statistically significant mean reliability differences for scores across the 5 scales, with the highest level of reliability observed for scores on the measure of Negative Emotionality and the lowest levels of reliability observed for scores on the measures of Aggression and Disconstraint. There was no evidence that the sex-composition of a sample was systematically related to score reliability, and there were no statistically significant differences in reliability between scores obtained with the English version of the test and those obtained with translated forms. However, reliability was consistently lower for scores on some scales when the data were obtained in nonclinical settings as opposed to clinical ones. Sample size was not significantly correlated with reliability estimates. RG methods have the potential for deepening the level of understanding about the role of reliability in the evaluation and use of personality tests.

ACKNOWLEDGMENTS

This project could not have been completed without the generosity of several people. I am grateful to Paul Arbisi, Jim Butcher, Isabelle Gillet, Roger Greene, Gloria Leon, Barbara Long, Dubi Lufi, John McNulty, and Nathan Weed, each of whom granted me access to their preexisting data sets so that I could calculate the necessary alpha coefficients. In addition, I am grateful to Moshe Almagor, Fanny Cheung, Jan Derksen, Kyunghee Han, Zhang Jianxin, Huub Schaeks, and Saulo Sirigatti, each of whom were instrumental in making previously calculated alpha coefficients available to me or who calculated alpha coefficients for me with their preexisting data sets. I would also like to thank two anonymous reviewers for invaluable comments on previous versions of this article.

Notes

^a The “test” refers to the Personality Psychopathology Five.

^a n = 30.

^b n = 29.

^a n = 45.

^b n = 18.

^a n = 46.

^b n = 17.