A Bayesian Approach to More Stable Estimates of Group-Level Effects in Contextual Studies

Abstract

Multilevel analyses are often used to estimate the effects of group-level constructs. However, when using aggregated individual data (e.g., student ratings) to assess a group-level construct (e.g., classroom climate), the observed group mean might not provide a reliable measure of the unobserved latent group mean. In the present article, we propose a Bayesian approach that can be used to estimate a multilevel latent covariate model, which corrects for the unreliable assessment of the latent group mean when estimating the group-level effect. A simulation study was conducted to evaluate the choice of different priors for the group-level variance of the predictor variable and to compare the Bayesian approach with the maximum likelihood approach implemented in the software Mplus. Results showed that, under problematic conditions (i.e., small number of groups, predictor variable with a small ICC), the Bayesian approach produced more accurate estimates of the group-level effect than the maximum likelihood approach did.

KEYWORDS:

• Bayesian estimation
• multilevel modeling
• structural equation modeling
• latent covariate model
• contextual analysis

Notes

In practice, the applied researcher would prefer to run more iterations when using MCMC methods. However, for the 432,000 Bayesian analyses in the simulation, it would not have been practical to run longer chains (see also Lambert et al., 2005). Running the simulation in parallel on a high-performance computer (two 2.9 GHz processors, each with eight cores) took about 5 weeks. In the illustrative example, we used much longer chains.

In additional analyses, we also investigated the statistical behavior of the estimates of the within-group coefficient. Tables S4 to S6 in the Supplemental Material show the estimated relative bias, the estimated relative RMSE, and the estimated coverage rate as a function of number of groups, ICC of the predictor variable, group size, and estimation method. With an absolute value of the estimated relative bias below.05, all estimators were approximately unbiased across conditions. In addition, the estimators did not differ with respect to the estimated relative RMSE, and the coverage values were all close to the nominal 95% level.

The sample used in the present analysis was a subsample drawn randomly from the original study sample (see Lüdtke et al., 2008). To illustrate differences in the results, we reduced the number of classes and the number of students within classes.

However, this also raises the question whether the estimates of the group-level effect could be improved even further by selecting larger and larger shape and scale parameters for the IG prior of the group-level variance. To further address this issue, we conducted a small simulation study (see Table S15 in the Supplemental Material) with J = 200, n = 5, , and several different shape/scale parameters for the IG prior: (.001, .001), (.01, .01), (.1, .1), (1, 1), (10, 10), (100, 100), and (1, 000, 1, 000). The results show that there is a turning point for further increasing the shape/scale and that with a shape/scale parameter of (1, 1) and larger, the bias introduced by the large shape/scale parameter outweighs the gains in efficiency (i.e., less variable estimates of the group-level effect). Thus, increasing the shape/scale parameter will not generally result in more accurate estimates of the group-level effect in terms of the RMSE. It is difficult to give general recommendations for the IG prior because the effect on the RMSE of changing the shape and scale will depend on the number of groups, the ICC of the predictor, and the group size.