Practical Applications of Generalizability Theory for Designing, Evaluating, and Improving Psychological Assessments

ABSTRACT

In this article, we illustrate how generalizability theory (G-theory) can extend traditional assessment methods for designing, improving, and evaluating results from both objectively and subjectively scored measures of individual differences. Our illustrations include quantification of multiple sources of measurement error, derivation of unique indexes of consistency for norm- and criterion-referenced interpretations of scores, estimation of score consistency when changing a measurement procedure, and disattenuation of correlation coefficients for measurement error. We also expand G-theory analyses beyond the item level to include parcels and split measures and highlight linkages among G-theory, classical test theory, and structural equation modeling. Computer code and sample data are provided in online supplements to help readers apply the demonstrated techniques to their own assessments.

Acknowledgment

We thank Professor Herbert Marsh for granting us permission to include data from the General Self scale of the Self-Description Questionnaire III (Marsh, 1992) as online supplements.

Notes

¹ Although we make no such distinctions here, standard practice in treatments of G-theory is to use ns without primes for sample sizes in G-studies and ns with primes for sample sizes in D-studies (see, e.g., Brennan, 2001; Cronbach et al., 1972).