Analyzing and Comparing Univariate, Multivariate, and Bifactor Generalizability Theory Designs for Hierarchically Structured Personality Traits

Abstract

We demonstrate how to use structural equation models to represent generalizability theory-based univariate, multivariate, and bifactor model designs. Analyses encompassed multi-occasion data obtained from the recently expanded form of the Big Five Inventory (BFI-2) that measures the broad personality domain constructs Agreeableness, Conscientiousness, Extraversion, Negative Emotionality, and Open-Mindedness along with three nested subdomain facets within each global domain. Results overall highlighted the importance of taking both item and occasion effects into account but underscored additional benefits of the multivariate and bifactor designs in providing more appropriate indices of generalizability for composite scores and effective ways to gauge subscale added value. Bifactor models further extended partitioning of universe score variance to separate general and group factor effects at both composite and subscale levels, expanded score consistency indices to distinguish or combine such effects, and allowed for further evaluation of score dimensionality and subscale viability. We provide guidelines, formulas, and code in R for analyzing all illustrated designs within the article and extended online Supplemental Material.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 Setting factor loadings, uniquenesses, and/or variances equal in GT-SEMs reflects assumptions of randomly parallel measures that allow for generalization of results to broader domains of items and occasions.

2 Although not discussed here, van Bork et al. (2022) further propose a theory of causal error and discuss its interrelationships with error defined in GT and latent state-trait theory.

3 Initial weights for subscales are usually based on the proportion of items for each subscale represented in the composite. However, the actual final weight for a subscale within indices for the composite also would depend on the values of variances and covariances for subscale scores. In multivariate GT, these weights are called effective weights (ew; Brennan, 2001, pp. 306-307).

4 Formal notation for multivariate GT designs is extended to indicate whether persons and facets are crossed with or nested under subscales in the design. Closed circles are used when persons or facets are crossed with subscales, whereas open circles are used when they are nested. The pi multivariate design for the top model in Figure 2 would be formally labeled as $p^{•}\times i^{\circ }$ because persons are crossed with subscales, and items are nested within subscales. The pio multivariate design for the bottom model in Figure 2 would be formally labeled as $p^{•}\times i^{\circ }\times o^{•}$ because persons and occasions are crossed with subscales and items are nested within subscales (see Brennan, 2001, Chapters 9 and 10 for further details).

5 We tested the best fitting confirmatory correlated factor models from Soto and John (2017) using WLSMV estimation in R and obtained excellent fits within each personality domain (CFIs: 0.97-1.00, TLIs: 0.95-1.00, RMSEA: 0.03-0.06). In absolute value, correlations between composite scores ranged from 0.07 to 0.42, and those between subscale scores within domains on average exceeded those across domains (M = 0.56 vs 0.18; see online Supplemental Materials).