Model Selection and Model Averaging for Mixed-Effects Models with Crossed Random Effects for Subjects and Items

Abstract

A good deal of experimental research is characterized by the presence of random effects on subjects and items. A standard modeling approach that includes such sources of variability is the mixed-effects models (MEMs) with crossed random effects. However, under-parameterizing or over-parameterizing the random structure of MEMs bias the estimations of the Standard Errors (SEs) of fixed effects. In this simulation study, we examined two different but complementary perspectives: model selection with likelihood-ratio tests, AIC, and BIC; and model averaging with Akaike weights. Results showed that true model selection was constant across the different strategies examined (including ML and REML estimators). However, sample size and variance of random slopes were found to explain true model selection and SE bias of fixed effects. No relevant differences in SE bias were found for model selection and model averaging. Sample size and variance of random slopes interacted with the estimator to explain SE bias. Only the within-subjects effect showed significant underestimation of SEs with smaller number of items and larger item random slopes. SE bias was higher for ML than REML, but the variability of SE bias was the opposite. Such variability can be translated into high rates of unacceptable bias in many replications.

Keywords:

• Mixed-effects models
• crossed random effects
• random slopes
• model selection
• model averaging
• ML
• REML

Article information

Conflict of Interest Disclosures: Each author signed a form for disclosure of potential conflicts of interest. No authors reported any financial or other conflicts of interest in relation to the work described.

Ethical Principles: The authors affirm having followed professional ethical guidelines in preparing this work. These guidelines include obtaining informed consent from human participants, maintaining ethical treatment and respect for the rights of human or animal participants, and ensuring the privacy of participants and their data, such as ensuring that individual participants cannot be identified in reported results or from publicly available original or archival data.

Funding: This work was not supported.

Role of the Funders/Sponsors: None of the funders or sponsors of this research had any role in the design and conduct of the study; collection, management, analysis, and interpretation of data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

Acknowledgments: The ideas and opinions expressed herein are those of the authors alone, and endorsement by the authors’ institutions is not intended and should not be inferred.

Notes

1 In some contexts, fixed and random effects of MEMs are called models for the mean and variance, respectively.

2 Note that random effects are relevant for different reasons. One of them is their relevance to estimate SEs and p-values of fixed effects. Other more common reason is they provide information about variance in the data. Not only the presence of relevant random effects can be useful, but can be the focus of hypothesis testing. This aspect is particularly relevant because the parameterization of random effects should be considered a confirmatory hypothesis testing itself (Barr, 2013).

3 Given that the effect size can be influenced by random slopes variances, two different versions of the effect size were tested in the present study. The first one was the simulated effect size (Table 1). The second one was an effect size that was corrected by the sampling variance considering the error term and the size of the random slopes of subjects and items. Willett (1989) presents two different illustrations for longitudinal modeling, and Judd et al. (2017) present a similar approach for experimental research. In the present study, no relevant differences were found between these two approaches when evaluating the influence of effect size in true model selection and SE bias. Thus, we report the results of the original effect size from Table 1.

4 We called it minimal model following Matuschek et al. (2017).

5 We called it maximal model following Barr et al. (2013).

6 We also analyzed the performance of AICc correcting for the sample size (number of subjects) and the number of observations (number of subjects multiplied by number of items), but no relevant differences in true model selection were obtained with AIC index. For the sake of brevity, we decided to focus on AIC and BIC indices.