Examining the Rule of Thumb of Not Using Multilevel Modeling: The “Design Effect Smaller Than Two” Rule

Abstract

Educational researchers commonly use the rule of thumb of “design effect smaller than 2” as the justification of not accounting for the multilevel or clustered structure in their data. The rule, however, has not yet been systematically studied in previous research. In the present study, we generated data from three different models (which differ in the location of the clustering effect). With a 3 (design effect) × 5 (cluster size) × 4 (number of clusters) Monte Carlo simulation study we found that the rule should not be applied when researchers: (a) are interested in the effects of higher-level predictors, or (b) have a cluster size less than 10. Implications of the findings and limitations of the study are discussed.

Keywords:

• clustering
• design effects
• intraclass correlation
• multilevel
• simulation studies

Notes

¹As discussed in Greene (2003, chapter 17), in multiple regression analyses when the normality assumption of the errors holds, as is the case for the data generating models of the present study, the ordinary least squares estimates and the maximum likelihood estimates are equivalent.

Additional information

Notes on contributors

Mark H. C. Lai

Mark H. C. Lai is a Ph.D. student at the Department of Educational Psychology, Texas A&M University. His research interests are effect size measures in multilevel modeling and evaluating measurement invariance using structural equation modeling.

Oi-man Kwok

Oi-man Kwok is an associate professor at the Department of Educational Psychology, Texas A&M University. His main research interests include modeling longitudinal data using structural equation models and multilevel models.