Using Parallel Splits with Self-Report and Other Measures to Enhance Precision in Generalizability Theory Analyses

Abstract

Although generalizability theory (G-theory) provides indices of reliability that take multiple sources of measurement error into account, those indices are typically conservative in nature because they reflect random rather than classical parallelism. One way to address these shortcomings is to use parallel splits rather than items as the unit of analysis in G-theory designs. In this article, we provide the most extensive treatment to date in how to effectively integrate parallel splits into an extended set of G-theory designs using data from the newly developed version of the Big Five Inventory (BFI-2; Soto & John). Results revealed that properly designed splits approximated classical parallelism while improving overall score consistency and reducing key components of measurement error. Variance components within appropriately chosen G-theory designs also provided effective means to evaluate the quality of splits and determine the best ways to improve score consistency and reduce specific sources of measurement error. To help readers in applying these techniques, we provide a comprehensive instructional supplement with code in R for creating parallel splits, analyzing all illustrated designs, and modifying those designs for other objectively or subjectively scored measures.

Acknowledgments

We thank the Iowa Measurement Research Foundation for providing research assistantship support for this project.