Bayesian Maximal Reliability Evaluation Using Latent Variable Modeling

ABSTRACT

We discuss an application of Bayesian factor analysis for estimation of the optimal linear combination and associated maximal reliability of a multi-component measuring instrument. The described procedure yields point and credibility interval estimates of this reliability coefficient, which are readily obtained in educational and behavioral measurement research. In addition, the outlined method permits evaluation of the gain in measurement consistency resulting from utilizing the maximal reliability coefficient instead of the traditionally used overall sum score reliability. The discussed Bayesian inference approach is applicable with widely available software in empirical studies, and is illustrated using a data example.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Author Note

We thank S. Depaoli for valuable discussions on Bayesian statistics, its applications, and software implementation. We are grateful to T. Patelis and two anonymous Referees for critical comments on an earlier version of the paper, which have contributed substantially to its improvement.

Notes

1 As would be seen when replicating the analyses reported in this section, and from the pertinent source code and output sections (see Appendix 2), we use default – and in effect non-informative – priors that are as follows (see Muthén & Muthén, 2024, ch. 12). First, normal priors with mean 0 and very large variance (viz. 10¹⁰) are utilized for the factor loadings and mean intercepts. Second, Inverse Gamma distributions are used for the unique variances, with parameters −1 and 0. (The discussion and conclusion section as well as Footnote 3 provide specific additional motivation for the choice of these priors.) At the software level, the default convergence cutoff of .05 was also utilized for the R-hat values (potential scale reduction factors, abbr. PSRFs). All parameters were thereby found to converge (with the PSRFs below 1.05 for all 10 model parameters (see also Muthén, Muthén, & Asparouhov, 2016; Gelman et al., 2013; Depaoli, 2021; and the second-last output section when replicating the section results).

2 With the software employed for the aims of the illustration section, half of the requested iterations – that is, 100,000 - were utilized as burn-in iterations within each of the two chains used (cf. Raykov, Marcoulides, & Schumacker, in press). For each parameter, like mentioned in Footnote 1, convergence was assessed using (i) the Gelman-Rubin convergence criterion based on the PSRF indices that early in the iterations history fell under 1.1 and remained below 1.1 for the rest of the 200,000 iterations, and this process was aided by examining (ii) the trace plots of the posterior draws in the chains as well as (iii) the auto-correlation plots. (All these plots and following results are obtained when using the source codes in Appendices 1 and 2.) In the process of this convergence assessment, both chains were accordingly found to mix well and hence the default of 1 iteration was used for thinning (i.e., each iteration was used for the analytic purposes of the illustration section after the burn-in iterations). We would also like to stress, as mentioned in the main text, that the overall model fit was assessed by posterior predictive checking (PPC). In PPC, the posterior predictive distribution is compared to the observed data, and as implemented in the used software, it is based on the standard likelihood-ratio chi-square statistic that compares sample and model-estimated means, variances, and covariances (Muthén, Muthén, & Asparouhov, 2016; cf., p. 400; Gelman et al., 2013). An essential element of the PPC then is the use of the observed-replicated chi-square differences confidence interval, which is also reported in the main text. The underlying p-value mentioned there as well (often referred to as PPP-value), was computed by the software based on the likelihood-ratio chi-square statistic. Further details regarding the numerical model fitting and parameter estimation are found in Muthén and Muthén (2024, ch. 9; see also Depaoli, 2021; Gelman et al., 2013; Muthén, Muthén, & Asparouhov, 2016, pp. 400–401).

3 Our choice to use non-informative priors in the illustration section was also informed by the fact that there are in general infinitely many possible informative priors for a given parameter. For this reason, in our view an appropriate choice among them is better carried out by substantive experts in a domain of application collaborating with Bayesian statisticians. This may best be done after a thorough elicitation of parameter prior distributions, which is to be grounded in extant research. Last but not least, we also employed (essentially) non-informative priors in order not to be suggestive of particular priors to possible users of the outlined procedure. (For the alternative case, Note 2 to Appendix 2 outlines how to utilize informative priors with the used software.)