Examining the Rule of Thumb of Not Using Multilevel Modeling: The “Design Effect Smaller Than Two” Rule

REFERENCES

• Beretvas, S.N. (2010). Cross-classified and multiple-membership models. In J.J. Hox & J.K. Roberts (Eds.), Handbook of advanced multilevel analysis (pp. 313–334). New York, NY: Routledge.
   Google Scholar
• Bonnet, M., Goossens, F.A., & Schuengel, C. (2011). Parental strategies and trajectories of peer victimization in 4 to 5 year olds. Journal of School Psychology, 49, 385–398. doi: 10.1016/j.jsp.2011.04.002
   PubMed Web of Science ®Google Scholar
• Bouman, T., van der Meulen, M., Goossens, F.A., Olthof, T., Vermande, M.M., & Aleva, E.A. (2012). Peer and self-reports of victimization and bullying: Their differential association with internalizing problems and social adjustment. Journal of School Psychology, 50, 759–774. doi: 10.1016/j.jsp.2012.08.004
   PubMed Web of Science ®Google Scholar
• Browne, W.J., & Draper, D. (2000). Implementation and performance issues in the Bayesian and likelihood fitting of multilevel models. Computational Statistics, 15, 391–420. doi: 10.1007/s001800000041
   Web of Science ®Google Scholar
• Clarke, P. (2008). When can group level clustering be ignored? Multilevel models versus single-level models with sparse data. Journal of Epidemiology and Community Health, 62, 752–758. doi: 10.1136/jech.2007.060798
   PubMed Web of Science ®Google Scholar
• Corte, C., & Zucker, R.A. (2008). Self-concept disturbances: Cognitive vulnerability for early drinking and early drunkenness in adolescents at high risk for alcohol problems. Addictive Behaviors, 33, 1282–1290. doi: 10.1016/j.addbeh.2008.06.002
   PubMed Web of Science ®Google Scholar
• De Los Reyes, A., Youngstrom, E.A., Swan, A.J., Youngstrom, J.K., Feeny, N.C., & Findling, R.L. (2011). Informant discrepancies in clinical reports of youths and interviewers’ impressions of the reliability of informants. Journal of Child and Adolescent Psychopharmacology, 21, 417–24. doi: 10.1089/cap.2011.0011
   PubMed Web of Science ®Google Scholar
• Deng, S., Lopez, V., Roosa, M.W., Ryu, E., Burrell, G.L., Tein, J.-Y., & Crowder, S. (2006). Family processes mediating the relationship of neighborhood disadvantage to early adolescent internalizing problems. The Journal of Early Adolescence, 26, 206–231. doi: 10.1177/0272431605285720
   Web of Science ®Google Scholar
• Enders, C.K., & Tofighi, D. (2007). Centering predictor variables in cross-sectional multilevel models: A new look at an old issue. Psychological Methods, 12, 121–138. doi: 10.1037/1082-989X.12.2.121
   PubMed Web of Science ®Google Scholar
• Fuentes, M., Hart-Johnson, T., & Green, C.R. (2007). The association among neighborhood socioeconomic status, race and chronic pain in Black and White older adults. Journal of the National Medical Association, 99, 1160–1169.
   PubMed Web of Science ®Google Scholar
• Goldstein, H., & Spiegelhalter, D.J. (1996). League tables and their limitations: Statistical issues in comparisons of institutional performance. Journal of the Royal Statistical Society. Series A (Statistics in Society), 159, 385–443. doi: 10.2307/2983325
   Web of Science ®Google Scholar
• Greene, W.H. (2003). Econometric analysis (5th ed.). Upper Saddle River, NJ: Prentice Hall.
   Google Scholar
• Hong, S., & You, S. (2012). Understanding Latino children's heterogeneous academic growth trajectories: Latent growth mixture modeling approach. The Journal of Educational Research, 105, 235–244. doi: 10.1080/00220671.2011.584921
   Web of Science ®Google Scholar
• Hoogland, J.J., & Boomsma, A. (1998). Robustness studies in covariance structure modeling: An overview and a meta-analysis. Sociological Methods & Research, 26, 329–367. doi: 10.1177/0049124198026003003
   Web of Science ®Google Scholar
• Hox, J.J. (2010). Multilevel analysis: Techniques and applications (2nd ed.). New York, NY: Routledge.
   Google Scholar
• Hox, J.J., & Maas, C.J. M. (2002). Sample sizes for multilevel modeling. In J. Blasius, J.J. Hox, E. de Leeuw, & P. Schmidt (Eds.), Social science methodology in the new millennium: Proceedings of the fifth international conference on logic and methodology (2nd expanded ed.). Opladen, RG: Leske + Budrich.
   Google Scholar
• Jester, J.M., Nigg, J.T., Buu, A., Puttler, L.I., Glass, J.M., Heitzeg, M.M., …, & Zucker, R.A. (2008). Trajectories of childhood aggression and inattention/hyperactivity: Differential effects on substance abuse in adolescence. Journal of the American Academy of Child and Adolescent Psychiatry, 47, 1158–1165. doi: 10.1097/CHI.0b013e3181825a4e
   PubMed Web of Science ®Google Scholar
• Kilian, B., Hofer, M., & Kuhnle, C. (2010). Value orientations as determinants and outcomes of conflicts between on-task and off-task actions in the classroom. Learning and Individual Differences, 20, 501–506. doi: 10.1016/j.lindif.2010.03.003
   Web of Science ®Google Scholar
• Kish, L. (1965). Survey sampling. New York, NY: Wiley.
   Google Scholar
• Kreft, G.G., & de Leeuw, E.D. (1988). The see-saw effect: A multilevel problem? Quality and Quantity, 22, 127–137. doi: 10.1007/bf00223037
   Web of Science ®Google Scholar
• Kwok, O., West, S.G., & Green, S.B. (2007). The impact of misspecifying the within-subject covariance structure in multiwave longitudinal multilevel models: A Monte Carlo study. Multivariate Behavioral Research, 42, 557–592. doi: 10.1080/00273170701540537
   Web of Science ®Google Scholar
• LaVange, L.M., Stearns, S.C., Lafata, J.E., Koch, G.G., & Shah, B.V. (1996). Innovative strategies using SUDAAN for analysis of health surveys with complex samples. Statistical Methods in Medical Research, 5, 311–329. doi: 096228029600500306
   PubMedGoogle Scholar
• Linnenbrink-Garcia, L., Rogat, T.K., & Koskey, K.L. K. (2011). Affect and engagement during small group instruction. Contemporary Educational Psychology, 36, 13–24. doi: 10.1016/j.cedpsych.2010.09.001
   Web of Science ®Google Scholar
• Ly, J., Zhou, Q., Chu, K., & Chen, S.H. (2012). Teacher–child relationship quality and academic achievement of Chinese American children in immigrant families. Journal of School Psychology, 50, 535–553.
   PubMed Web of Science ®Google Scholar
• Maas, C.J. M., & Hox, J.J. (2004). The influence of violations of assumptions on multilevel parameter estimates and their standard errors. Computational Statistics & Data Analysis, 46, 427–440. doi: 10.1016/j.csda.2003.08.006
   Web of Science ®Google Scholar
• Maas, C.J. M., & Hox, J.J. (2005). Sufficient sample sizes for multilevel modeling. Methodology, 1, 86–92. doi: 10.1027/1614-1881.1.3.86
   Google Scholar
• Muthén, B.O. (1994). Multilevel covariance structure analysis. Sociological Methods & Research, 22, 376–398. doi: 10.1177/0049124194022003006
   Web of Science ®Google Scholar
• Muthén, B.O., & Satorra, A. (1995). Complex sample data in structural equation modeling. Sociological Methodology, 25, 267–316. doi: 10.2307/271070
   Web of Science ®Google Scholar
• Muthén, L.K., & Muthén, B.O. (1998–2012). Mplus user's guide (7th ed.). Los Angeles, CA: Authors.
   Google Scholar
• Peugh, J.L. (2010). A practical guide to multilevel modeling. Journal of School Psychology, 48, 85–112. doi: 10.1016/j.jsp.2009.09.002
   PubMed Web of Science ®Google Scholar
• Qureshi, I., & Fang, Y. (2011). Socialization in open source software projects: A growth mixture modeling approach. Organizational Research Methods, 14, 208–238.
   Web of Science ®Google Scholar
• Raudenbush, S.W., & Bryk, A.S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
   Google Scholar
• Rust, K.F., & Rao, J.N. K. (1996). Variance estimation for complex surveys using replication techniques. Statistical Methods in Medical Research, 5, 283–310. doi: 096228029600500305
   PubMedGoogle Scholar
• Semke, C.A., Garbacz, S.A., Kwon, K., Sheridan, S.M., & Woods, K.E. (2010). Family involvement for children with disruptive behaviors: The role of parenting stress and motivational beliefs. Journal of School Psychology, 48, 293–312. doi: 10.1016/j.jsp.2010.04.001
   PubMed Web of Science ®Google Scholar
• Snijders, T.A. B., & Bosker, R.J. (2012). Multilevel analysis: An introduction to basic and advanced multilevel modeling (2nd ed.). London, UK: Sage.
   Google Scholar
• Thomas, S.L., & Heck, R.H. (2001). Analysis of large-scale secondary data in higher education research: Potential perils associated with complex sampling designs. Research in Higher Education, 42, 517–540. doi: 10.1023/A:1011098109834
   Web of Science ®Google Scholar
• Turner, J.C., Midgley, C., Meyer, D.K., Gheen, M., Anderman, E.M., Kang, Y., & Patrick, H. (2002). The classroom environment and students’ reports of avoidance strategies in mathematics: A multimethod study. Journal of Educational Psychology, 94, 88–106. doi: 10.1037//0022-0663.94.1.88
   Web of Science ®Google Scholar
• Von Grünigen, R., Kochenderfer-Ladd, B., Perren, S., & Alsaker, F.D. (2012). Links between local language competence and peer relations among Swiss and immigrant children: The mediating role of social behavior. Journal of School Psychology, 50, 195–213. doi: 10.1016/j.jsp.2011.09.005
   PubMed Web of Science ®Google Scholar
• Wagner, U., Christ, O., Pettigrew, T.F., Stellmacher, J., & Wolf, C. (2006). Prejudice and minority proportion: Contact instead of threat effects. Social Psychology Quarterly, 69, 380–390. doi: 10.1177/019027250606900406
   Web of Science ®Google Scholar
• Wong, M.M., Nigg, J.T., Zucker, R.A., Puttler, L.I., Fitzgerald, H.E., Jester, J.M., …, & Adams, K. (2006). Behavioral control and resiliency in the onset of alcohol and illicit drug use: A prospective study from preschool to adolescence. Child Development, 77, 1016–1033. doi: 10.1111/j.1467-8624.2006.00916.x
   PubMed Web of Science ®Google Scholar