The Impact of Sample Size and Other Factors When Estimating Multilevel Logistic Models

REFERENCES

• Agresti, A. (2002). Categorical data analysis ( 2nd ed.). Hoboken, NJ: John Wiley & Sons.
   Google Scholar
• Agresti, A., Booth, J.G., Hobert, J.P., & Caffo, B. (2000). Random-effects modeling of categorical response data. Sociological Methodology, 30, 27–80. http://www.jstor.org/stable/271130
   Web of Science ®Google Scholar
• Aitkin, M., & Longford, N. (1986). Statistical modelling issues in school effectiveness studies. Journal of the Royal Statistical Society, Series A (General), 149, 1–43. doi: 10.2307/2981882
   Web of Science ®Google Scholar
• Albert, P.S. (1999). Longitudinal data analysis (repeated measures) in clinical trials. Statistics in Medicine, 18, 1707–1732. http://dx.doi.org/10.1081/SAC-200068364
   PubMed Web of Science ®Google Scholar
• Austin, P.C. (2005). Bias in penalized quasi-likelihood estimation in random effects logistic regression models when the random effects are not normally distributed. Communications in Statistics-Simulation and Computation, 34, 549–565. doi: 10.1081/SAC-200068364
   Web of Science ®Google Scholar
• Austin, P.C. (2007). A comparison of the statistical power of different methods for the analysis of cluster randomization trials with binary outcomes. Statistics in Medicine, 26, 3550–3565. doi: 10.1002/sim.2813
   PubMed Web of Science ®Google Scholar
• Austin, P.C. (2010). Estimating multilevel logistic regression models when the number of cluster is low: A comparison of different statistical software procedures. International Journal of Biostatistics, 6, 1–30. doi: 10.2202/1557-4679.1195
   Web of Science ®Google Scholar
• Bell, B.A., Morgan, G.B., Schoeneberger, J.A., Kromrey, J.D., & Ferron, J.M. (2014). How low can you go? An investigation of the influence of sample size and model complexity on point and interval estimates in two-level linear models. Methodology, 10
   Web of Science ®Google Scholar
• Bell, B.A., Schoeneberger, J.A., Morgan, G.B., Ferron, J.M., & Kromrey, J.D. (2010, April). N ≤ 30: Impact of small level-1 and level-2 sample sizes on estimates in two-level multilevel models. Presentation at the American Education Research Association Conference, Denver, CO.
   Google Scholar
• Bell, B.A., Schoeneberger, J.A., Morgan, G.B., Zhu, M., Ferron, J.M., & Kromrey, J.D. (2011, May). Hierarchical vs. contextual models: Sample size, model complexity, and the 30/30 rule. Presentation at the Annual Modern Modeling Methods (M3) Conference, Storrs, CT.
   Google Scholar
• Bellamy, S.L., Li, Y., Lin, X., & Ryan, L.M. (2005). Quantifying PQL bias in estimating cluster-level covariate effects in generalized linear models for group-randomized trials. Statistica Sinica, 15, 1015–1032. http://www3.stat.sinica.edu.tw/statistica
   Web of Science ®Google Scholar
• Bloom, H.S. (2005). Learning more from social experiments: Evolving analytic approaches. New York, NY: Russell Sage Foundation.
   Google Scholar
• Breslow, N. (2005). Whither PQL? Second Seattle Symposium in Biostatistics. New York, NY: Springer. http://www.bepress.com/cgi/viewcontent.cgi?article=1015&context=uwbiostat
   Google Scholar
• Breslow, N.E., & Clayton, D.G. (1993). Approximate inference in generalized linear models. Journal of the American Statistical Association, 88, 9–25. doi: 10.2307/2290687
   Web of Science ®Google Scholar
• Breslow, N.E., & Lin, X. (1995). Bias correction in generalised linear mixed models with a single component of dispersion. Biometrika, 82, 81–91. doi: 10.1093/biomet/82.1.81
   Web of Science ®Google Scholar
• Browne, W.J., & Draper, D. (2000). Implementation and performance issues in the Bayesian and likelihood fitting of multilevel models. Computational Statistics, 15, 391–420. doi: 10.1007/s001800000041
   Web of Science ®Google Scholar
• Browne, W.J., Lahi, M.G.,& Parker, R.M. (2009). A Guide to Sample Size Calculations for Random Effect Models via Simulation and the MLPowSim Software Package [Computer program and manual]. Retrieved from http://www.bristol.ac.uk/cmm/software/mlpowsim/
   Google Scholar
• Burton, A., Altman, D.G., Royston, P., & Holder, R.L. (2006). The design of simulation studies in medical statistics. Statistics in Medicine, 25, 4279–4292. doi: 10.1002/sim.2673
   PubMed Web of Science ®Google Scholar
• Callens, M., & Croux, C. (2005). Performance of likelihood-based estimation methods for multilevel binary regression models. Journal of Statistical Computation and Simulation, 75, 1003–1017. doi: 10.1080/00949650412331321070
   Web of Science ®Google Scholar
• Clarke, P. (2008). When can group level clustering be ignored? Multilevel models versus single-level models with sparse data. Journal of Epidemiology and Community Health, 62, 752–758. doi: 10.1136/jech.2007.060798
   PubMed Web of Science ®Google Scholar
• Clarke, P. & Wheaton, B. (2007). Addressing data sparseness in contextual population research: Using cluster analysis to create synthetic neighborhoods. Sociological Methods & Research, 35, 311–351. doi: 10.1177/0049124106292362
   Web of Science ®Google Scholar
• Cohen, J. (1983). The cost of dichotomization. Applied Psychological Measurement, 7, 249–253. doi: 10.1177/014662168300700301
   Web of Science ®Google Scholar
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences ( 2nd ed.). New York, NY: Psychology Press.
   Google Scholar
• De Leeuw, J., & Kreft, I.G. (1998). Introducing multilevel modeling. Newbury Park, CA: Sage.
   Google Scholar
• Diaz, R.E. (2007). Comparison of PQL and Laplace 6 estimates of hierarchical linear models when comparing groups of small incident rates in cluster randromised trials. Computational Statistics & Data Analysis, 51, 2871–2888. doi: 10.1016/j.csda.2006.10.005
   Web of Science ®Google Scholar
• French, B.F., & Finch, W.H. (2010). Hierarchical logistic regression: Accounting for multilevel data in DIF detection. Journal of Educationl Measurement, 47, 299–317. doi: 10.1111/j.1745-3984.2010.00115.x
   Web of Science ®Google Scholar
• Gibbons, R.D., & Bock, R.D. (1987). Trend in correlated proportions. Psychometrika, 52, 113–124. doi: 10.1007/BF02293959
   Web of Science ®Google Scholar
• Goldstein, H. (1987). Multilevel models in educational and social research. London, UK: Griffin.
   Google Scholar
• Goldstein, H. (1991). Nonlinear multilevel models, with an application to discrete response data. Biometrika, 78, 45–51. doi: 10.1093/biomet/78.1.45
   Web of Science ®Google Scholar
• Goldstein, H. (2003). Multilevel statistical models. ( 3rd ed.). London, UK: Arnold.
   Google Scholar
• Goldstein, H., & Rasbash, J. (1996). Improved approximations for multilevel models with binary responses. Journal of the Royal Statistical Society. Series A (Statistics in Society), 159, 505–513. doi: 10.2307/2983328
   Web of Science ®Google Scholar
• Gradstein, M. (1986). Maximal correlation between normal and dichotomous variables. Journal of Educational Statistics, 11, 259–261. doi: 10.2307/1164698
   Google Scholar
• Gregoire, T., & Schabenberger, O. (1996). A non-linear mixed-effects model to predict cumulative bole volume of standing trees. Journal of Applied Statistics, 23(2–3), 257–272. doi: 10.1080/02664769624233
   Web of Science ®Google Scholar
• Guo, G., & Zhao, H. (2000). Multilevel modeling for binary data. Annual Review of Sociology, 26, 441–462. doi: 10.1146/annurev.soc.26.1.441
   Web of Science ®Google Scholar
• Hox, J.J. (2010). Multilevel Analysis: Techniques and Applications. New York, NY: Routledge.
   Google Scholar
• Jang, W., & Lim, J. (2009). A numerical study of PQL estimation biases in generalized linear mixed models under heterogeneity of random effects. Communications in Statistics—Simulation and Computation, 38, 692–702. doi: 10.1080/03610910802627055
   Web of Science ®Google Scholar
• Kim, Y., Choi, Y. & Emery, S. (2013). Logistic regression with multiple random effects: A simulation study of estimation methods and statistical packages. American Statistician, 67, 171–182. doi: 10.1080/00031305.2013.817357
   Web of Science ®Google Scholar
• Kreft, I.G., & De Leeuw, J. (1998). Introducing multilevel modeling. Newbury Park, CA: Sage.
   Google Scholar
• Li, Y. (2006). Power analysis for a mixed effects logistic regression model [Dissertation]. Baton Rouge, LA: Louisiana State University.
   Google Scholar
• Lin, X., & Breslow, N.E. (1996). Bias correction in generalized linear mixed models with multiple components of dispersion. Journal of the American Statistical Association, 91, 1007–1016. doi: 10.2307/2291720
   Web of Science ®Google Scholar
• Lindstrom, M. & Bates, D. (1990). Nonlinear mixed effects models for repeated measures data. Biometrics, 46, 673–687. doi: 10.2307/2532087
   PubMed Web of Science ®Google Scholar
• Longford, N.T. (1993). Random coefficient models. Oxford, UK: Clarendon Press.
   Google Scholar
• Longford, N.T. (1994). Logistic regression with random coefficients. Computational Statistics & Data Analysis, 17, 1–15. doi: 10.1016/0167-9473(92)00062-V
   Web of Science ®Google Scholar
• Maas, C.J., & Hox, J.J. (2004). Robustness issues in multilevel regression analysis. Statistica Neerlandica, 58, 127–137. doi: 10.1046/j.0039-0402.2003.00252.x
   Web of Science ®Google Scholar
• Maas, C.J., & Hox, J.J. (2005). Sufficient sample sizes for multilevel modeling. Methodology, 1, 86–92. doi: 10.1027/1614-2241.1.3.85
   Google Scholar
• Maxwell, S.E., & Delaney, H.D. (1993). Bivariate median splits and spurious statistical significance. Psychological Bulletin, 113, 181–190. doi: 10.1037/0033-2909.113.1.181
   Web of Science ®Google Scholar
• Maxwell, S.E., & Delaney, H.D. (2004). Designing experiments and analyzing data: A model comparison perspective ( 2nd ed.). Mahwah, NJ: Lawrence Erlbaum Associates.
   Google Scholar
• McCoach, D.B. & Black, A.C. (2008). Evaluation of model fit and adequacy. In D.B. McCoach & A.C. Black (Eds.), Multilevel modeling of educational data (pp. 245–272). Charlotte, NC: Information Age.
   Google Scholar
• McCullagh, P., & Nelder, J.A. (1989). Generalized linear models ( 2nd ed.). London, UK: Chapman & Hall.
   Google Scholar
• McCulloch, C.E., Searle, S.R., & Newhaus, J.M. (2008). Generalized, linear, and mixed models. Hoboken, NJ: John Wiley & Sons.
   Google Scholar
• Moineddin, R., Matheson, F.I., & Glazier, R.H. (2007). A simulation study of sample size for multilevel logistic regression. BMC Medical Research Methodology, 7(34), 1–10. doi: 10.1186/1471-2288-7-34
   PubMed Web of Science ®Google Scholar
• Mok, M. (1995). Sample size requirements for 2-level designs in educational research [Unpublished manuscript]. Sydney, Australia: Macquarie University.
   Google Scholar
• Murnane, R.J., & Willett, J.B. (2011). Methods matter: Improving causal inference in educational and social science research. New York, NY: Oxford University Press.
   Google Scholar
• Pacagnella, O. (2011). Sample size and accuracy of estimates in multilevel models: New simulation results. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 7, 111–120. doi: 10.1027/1614-2241/a000029
   Web of Science ®Google Scholar
• Pedhazur, E.J. (1997). Multiple regression in behavioral research ( 3rd ed.). Fort Worth, TX: Harcourt Brace College.
   Google Scholar
• Raftery, A.E. (1995). Bayesian model selection in social research. Sociological Methodology, 25, 111–163. doi: 10.2307/271063
   Web of Science ®Google Scholar
• Raudenbush, S.W., & Bryk, A.S. (1986). A hierarchical model for studying school effects. Sociology of Education, 59, 1–17. doi: 10.2307/2112482
   Web of Science ®Google Scholar
• Raudenbush, S.W., & Bryk, A.S. (2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage.
   Google Scholar
• Raudenbush, S.W., Spybrook, J., Liu, X., & Congdon, R. (2005). Optimal design for longitudinal and multilevel research (Version 1.55) [Computer software]. Chicago, IL: University of Chicago Press.
   Google Scholar
• Raudenbush, S.W., Yang, M.-L., & Yosef, M. (2000). Maximum likelihood for generalized linear models with nested random effects via high-order, multivariate Laplace apprxomation. Journal of Computational and Graphical Statistics, 9, 141–157. doi: 10.2307/1390617
   Web of Science ®Google Scholar
• Rodriguez, G., & Goldman, N. (1995). An assessment of estimatin procedures for multilevel models with binary responses. Journal of the Royal Statistical Society. Series A (Statistics in Society), 158, 73–89. doi: 10.2307/2983404
   Web of Science ®Google Scholar
• Rodriguez, G., & Goldman, N. (2001). Improved estimation procedures for multilevel models with binary response: A case-study. Journal of the Royal Statistical Society. Series A (Statistics in Society), 164, 339–355. doi: 10.1111/1467-985X.00206
   Web of Science ®Google Scholar
• Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. doi: 10.1037/0033-2909.86.3.638
   Web of Science ®Google Scholar
• SAS Institute, Inc. (2008). SAS/IML® 9.2 user's guide. Cary, NC: SAS Institute, Inc.
   Google Scholar
• Schall, R. (1991). Estimation in generalized linear models with random effects. Biometrika, 78, 719–727. doi: 10.1093/biomet/78.4.719
   Web of Science ®Google Scholar
• Sheiner, L.B. & Beal, S.L. (1980). Evaluation of methods for estimating population pharmacokinetic parameters. I. Michealis–Menton model: Routine clinical pharmacokinetic data. Journal of Pharmacokinetics and Biopharmaceutics, 8, 553–571. doi: 10.1007/BF01060053
   PubMedGoogle Scholar
• Shun, Z. (1997). Another look at the salamander mating data: A modified Laplace approximation approach. Journal of the American Statistical Association, 92, 341–349. doi: 10.2307/2291479
   Web of Science ®Google Scholar
• Shun, Z., & McCullagh, P. (1995). Laplace approximation of high dimensional integrals. Journal of the Royal Statistical Society. Series B (Methodological), 57, 749–760. http://www.jstor.org/pss/2345941
   Web of Science ®Google Scholar
• Singer, J.D., & Willett, J.B. (2003). Applied longitudinal data analysis: Modeling change and event occurrence. Oxford, UK: Oxford University Press.
   Google Scholar
• Snijders, T.A., & Bosker, R.J. (1999). Multilevel analysis. Thousand Oaks, CA: Sage.
   Google Scholar
• Snijders, T.A., Bosker, R.J., & Guldemond, H. (1996). PINT user's manual (Version 1.6) [Computer program and manual]. Retrieved from http://stat.gamma.rug.nl/snijders/multilevel.htm
   Google Scholar
• Solomon, P.J., & Cox, D.R. (1992). Nonlinear component of variance models. Biometrika, 79, 1–11. doi: 10.1093/biomet/79.1.1
   Web of Science ®Google Scholar
• Stevens, J.P. (2009). Applied multivariate statistics for the social sciences ( 5th ed.). New York, NY: Routledge.
   Google Scholar
• Ten Have, T.R., & Localio, A.R. (1999). Empirical Bayes estimation of random effects parameters in mixed effects logistic regression models. Biometrics, 55, 1022–1029.
   PubMed Web of Science ®Google Scholar
• Theall, K.P., Scribner, R., Broyles, S., Yu, Q., Chotalia, J., Simonsen, N., … Carolin, B.P. (2011). Impact of small group size on neighbourhood influence in multilevel models. Journal of Epidemiology Community Health, 65, 688–695. doi:
   PubMed Web of Science ®Google Scholar
• Van der Leeden, R., Busing, F.M., & Meijer, E. (1997, April). Applications of bootstrap methods for two-level models. Presentation at the Multilevel Conference, Amsterdam, The Netherlands.
   Google Scholar
• Vonesh, E.F. (1996). A note on the use of Laplace's approximation for nonlinear mixed-effects models. Biometrika, 83, 447–452. doi: 10.1093/biomet/83.2.447
   Web of Science ®Google Scholar
• Vonesh, E.F., & Carter, R.L. (1992). Mixed-effects nonlinear regression for unbalanced repeated measures. Biometrics, 48, 1–17. doi: 10.2307/2532734
   PubMed Web of Science ®Google Scholar
• Wolfinger, R. (1993). Laplace's approximation for nonlinear mixed models. Biometrika, 80(4), 791–795. doi: 10.1093/biomet/80.4.791
   Web of Science ®Google Scholar
• Wolfinger, R., & Lin, X. (1997). Two Taylor-series approximation methods for nonlinear mixed models. Computational Statistics & Data Analysis, 25, 465–490. doi: 10.1016/S0167-9473(97)00012-1
   Web of Science ®Google Scholar
• Wolfinger, R., & O’Connell, M. (1993). Generalized linear mixed models: A pseudo-likelihood approach. Journal of Statistical Computations and Simulations, 48, 233–243. doi: 10.1080/00949659308811554
   Web of Science ®Google Scholar
• Wong, G.Y., & Mason, W.M. (1985). The hierarchical logistic regression model for multilevel analysis. Journal of the American Statistical Association, 80, 513–524. doi: 10.2307/2288464
   Web of Science ®Google Scholar
• Zeger, S.L., Liang, K.Y., and Albert, P.S. (1988). Models for longitudinal data: A generalized estimating equation approach. Biometrics, 44, 1049–1060. doi: 10.2307/2531734
   PubMed Web of Science ®Google Scholar