Problems of Domain Factors with Small Factor Loadings in Bi-Factor Models

Abstract

Many measurement designs produce domain factors with small variances and factor loadings. The current study investigates the cause, prevalence, and problematic consequences of such domain factors. We collected a meta-analytic sample of empirical applications, conducted a simulation study on statistical power and estimation precision, and provide a reanalysis of an empirical example. The meta-analysis shows that about a quarter of all standardized domain factor loadings is in the range of $-.2<\lambda <.2$ and about a third of all domains is measured by five or fewer indicators, resulting in small factor variances. The simulation study examines the associated difficulties concerning statistical power, trait recovery, irregular estimates, and estimation precision for a range of such realistic cases. The empirical example illustrates the challenge to develop measures that produce clearly interpretable domain factors. Study planning and interpretation need to take the (expected) sum of squared factor loadings per domain factor into account. This is relevant even if influences of domain factors are desired to be small, and equally applies to different model variants. We propose several strategies for how researchers may better unlock the bifactor model’s full potential and clarify its interpretation.

KEYWORDS:

• bi-factor model
• statistical power
• specific factors
• bi-factor(S-1) model

Author note

The authors made the following contributions. Nils Petras: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Visualization, Writing - original draft, Writing - review & editing; Thorsten Meiser: Conceptualization, Supervision, Writing - Review & Editing.

Article information

Conflict of Interest Disclosures: Each author signed a form for disclosure of potential conflicts of interest. No authors reported any financial or other conflicts of interest in relation to the work described.

Ethical Principles: The authors affirm having followed professional ethical guidelines in preparing this work. These guidelines include obtaining informed consent from human participants, maintaining ethical treatment and respect for the rights of human or animal participants, and ensuring the privacy of participants and their data, such as ensuring that individual participants cannot be identified in reported results or from publicly available original or archival data.

Funding: This work was supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - GRK 2277 “Statistical Modeling in Psychology”.

Role of the Funders/Sponsors: None of the funders or sponsors of this research had any role in the design and conduct of the study; collection, management, analysis, and interpretation of data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

Acknowledgments: The authors would like thank Celine Kumpf and Alicia Gernand for their help on collecting the meta-analysis data. We thank Celine Kumpf and Alicia Gernand for their help on collecting the meta-analysis data. We thank Marie Mundt for assisting in the literature search for the empirical example with open data. The ideas and opinions expressed herein are those of the authors alone, and endorsement by the authors’ institution or the German Research Foundation is not intended and should not be inferred.

Data availability statement

All code and data of the meta-analysis and simulation study, as well as the code generating the manuscript are available here: https://osf.io/qys8u/.

Notes

1 For a discussion of problems regarding the estimation of correlated domain factors in the S model see Markon (2019). Conceptually, a full set of positively correlated domain factors (= correlations between all indicators) and the general factor are to some degree redundant, leading to problems in both estimation and interpretation.

2 In Exploratory Factor Analysis (EFA) and Principal Component Analysis (PCA), the eigenvalues of the covariance matrix are used as a decision criterion for the number of factors to include. In PCA, these are the $S{S}_{\lambda }$ values of the unrotated components and in EFA this relationship holds approximately. Therefore, the effective inclusion criterion is usually near $S{S}_{\lambda }=1.$

3 Setting the factor variance instead of the first loading to 1 for model identification would prevent that, but most likely shift the problem to other parameters. Therefore, we considered this to be a problematic phenomenon.

4 The S*I-1 variant is not discussed here.

5 8 of 15 indicator reliabilities exceeded 0.948, model fit was almost perfect (TLI = 1.00, CFI = 1.00, RMSEA = 0.010), despite the diverse indicator content (Blanco et al., 2014), Table 3).

6 Another one had to be omitted due to irregular estimates. The error variance of an indicator variable was estimated to be impossibly large and negative, leading to uninterpretable results.

7 One indicator was excluded from analysis due to an impossible combination of reported standardized factor loadings (${\lambda }_s=$ 0.98, ${\lambda }_g=$ 0.59).

8 The correlation between domain factors in the S-1c model also affects the statistical power and estimation precision (Yuan et al., 2010), but was not varied beyond the distinction between S-1 and S-1c models. Higher correlations were shown to lead to both increases and decreases in standard errors for both loadings and factor variances in correlated-factor models depending on the other model parameters (Yuan et al., 2010, Table 3). It is unclear if such differences are substantial in bi-factor models and how they would proliferate to other factors in the model. As seen below, the difference in statistical power between the S-1 and S-1c model, which is essentially a large variation of a domain factor correlation (0 vs. 0.5), proved to be relatively inconsistent and unimportant in comparison to other factors.

9 For outcomes on a scale of 0 to 1, we considered linear models to be sufficient, because they detect the presence of monotonous effects, and their easily interpretable determination coefficient is able to roughly order them by importance. Binomial regression would not have offered an easy to interpret determination coefficient and a logit transform would have led to many infinity values due to observed relative frequencies of exactly 1.

10 The same plot, but with 0.95 quantiles (instead of medians) of the RMSE distributions is included in the supplementary materials.

11 The same absolute difference on the scale of λ_s is larger on the scale of ${\lambda }_s^2$ (indicator variance) for larger values of λ_s. The truth of this claim, therefore, depends on the scale.

12 Given that the S-1 model was proposed along with a change in the interpretation of η_g, one could also understand this as the consequence of a change in the meaning of η_g. The current work can only demonstrate the recovery of the original data-generating trait η_g, not the interpretability or reliability of the resulting factor score if the S-1 model is estimated.

13 The dataset provided by Dueber and Toland (2023) omits two indicators refering to satisfaction with management.

14 There are notable differences in the factor loading estimates between this model and the values reported by the original authors (Dueber & Toland, 2023, Figure 6) because they estimated a model for categorical data, as can be seen in their open code.

15 The unique proportion of lower-order factor variance (disturbance) in higher-order factor models is not indicator-specific.

16 An alternative is the index H for construct reliability (Hancock, 2001; see also Rodriguez et al., 2016) which is more straightforward but does not take the impact of η_g into account (cf. Figure 8).