Using Generalizability Theory to Disattenuate Correlation Coefficients for Multiple Sources of Measurement Error

ABSTRACT

Over the years, research in the social sciences has been dominated by reporting of reliability coefficients that fail to account for key sources of measurement error. Use of these coefficients, in turn, to correct for measurement error can hinder scientific progress by misrepresenting true relationships among the underlying constructs being investigated. In the research reported here, we addressed these issues using generalizability theory (G-theory) in both traditional and new ways to account for the three key sources of measurement error (random-response, specific-factor, and transient) that affect scores from objectively scored measures. Results from 20 widely used measures of personality, self-concept, and socially desirable responding showed that conventional indices consistently misrepresented reliability and relationships among psychological constructs by failing to account for key sources of measurement error and correlated transient errors within occasions. The results further revealed that G-theory served as an effective framework for remedying these problems. We discuss possible extensions in future research and provide code from the computer package R in an online supplement to enable readers to apply the procedures we demonstrate to their own research.

KEYWORDS:

• Measurement models
• modern test theory
• reliability
• validity generalizability theory

Article Information

Conflict of interest disclosures: Each author signed a form for disclosure of potential conflicts of interest. No authors reported any financial or other conflicts of interest in relation to the work described.

Ethical principles: The authors affirm having followed professional ethical guidelines in preparing this work. These guidelines include obtaining informed consent from human participants, maintaining ethical treatment and respect for the rights of human or animal participants, and ensuring the privacy of participants and their data, such as ensuring that individual participants cannot be identified in reported results or from publicly available original or archival data.

Funding: This work was not supported.

Role of the funders/sponsors: None of the funders or sponsors of this research had any role in the design and conduct of the study; collection, management, analysis, and interpretation of data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

Acknowledgments: The ideas and opinions expressed herein are those of the authors alone, and endorsement by the authors' institutions is not intended and should not be inferred. The authors thank the Iowa Measurement Research Foundation for providing research assistantship support for this project and Professor Herbert Marsh for granting them permission to provide sample data from the SDQ-III as a companion to their online supplement.

Notes

1 Either arithmetic or geometric means can be used in the denominators of Equations 6(6) and 7(7) . Arithmetic and geometric means will yield the same value for the disattenuated correlation if the same type of mean is used in calculating the CESs within the equations (see equations 43 and 44 in Vispoel, et al., 2018a). Using arithmetic means in G-theory applications allows task and occasion scores to be essentially tau-equivalent, but not necessarily classically parallel. Arithmetic and geometric means are equal when scale variances are equal.

2 We verified that Equation 6(6) yields results for disattenuated correlations identical to those obtained from mGENOVA for multivariate random facet, persons × tasks × occasions designs.

3 In other treatments of G-theory (see, e.g., Brennan, 2001; Cronbach, Gleser, Nanda, & Rajaratnam, 1972), G-coefficients are represented by the symbol Eρ², which emphasizes that they reflect random parallelism.

4 In other treatments of G-theory (see, e.g., Brennan, 2001; Brennan & Kane, 1977), D-coefficients are represented by the symbol Φ.