Inference Based on the Best-Fitting Model Can Contribute to the Replication Crisis: Assessing Model Selection Uncertainty Using a Bootstrap Approach

REFERENCES

• Bakker, M., & Wicherts, J. M. (2014). Outlier removal, sum scores, and the inflation of the Type I error rate in independent samples t tests: The power of alternatives and recommendations. Psychological Methods, 19, 409–427. doi:10.1037/met0000014
   PubMed Web of Science ®Google Scholar
• Bollen, K. A., Harden, J. J., Ray, S., & Zavisca, J. (2014). BIC and alternative Bayesian information criteria in the selection of structural equation models. Structural Equation Modeling, 21, 1–19.
   PubMed Web of Science ®Google Scholar
• Bollen, K. A., Ray, S., Zavisca, J., & Harden, J. J. (2012). A comparison of Bayes factor approximation methods including two new methods. Sociological Methods & Research, 41, 294–324. doi:10.1177/0049124112452393
   Web of Science ®Google Scholar
• Browne, M., & Cudeck, R. (1992). Alternative ways of assessing model fit. Sociological Methods & Research, 21, 230–258. doi:10.1177/0049124192021002005
   Web of Science ®Google Scholar
• Buja, A., & Stuetzle, W. (2006). Observations on bagging. Statistica Sinica, 16, 323–351.
   Web of Science ®Google Scholar
• Chwe, M. S. (2014, January 31). Scientific pride and prejudice. The New York Times. Retrieved from http://www.nytimes.com/2014/02/02/opinion/sunday/scientific-pride-and-prejudice.html
   Google Scholar
• Cudeck, R., & Browne, M. W. (1983). Cross-validation of covariance structures. Multivariate Behavioral Research, 18, 147–167. doi:10.1207/s15327906mbr1802_2
   PubMed Web of Science ®Google Scholar
• Efron, B. (2014). Estimation and accuracy after model selection. Journal of the American Statistical Association, 109, 991–1007. doi:10.1080/01621459.2013.823775
   PubMed Web of Science ®Google Scholar
• Erosheva, E. A., & Curtis, S. M. (2013). Dealing with rotational invariance in Bayesian confirmatory factor analysis ( Tech. Rep. No. 589). Seattle, WA: Department of Statistics University of Washington.
   Google Scholar
• Gustafsson, J.-E. (2001). Measurement from a hierarchical point of view. In H. I. Braun, D. N. Jackson, & D. E. Wiley (Eds.), The role of constructs in psychological and educational measurement (pp. 77–101). Mahwah, NJ: Erlbaum.
   Google Scholar
• Hastie, T., Tibshirani, R., & Friedman, J. H. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). New York, NY: Springer.
   Google Scholar
• Hohensinn, C., Kubinger, K. D., & Reif, M. (2014). On robustness and power of the likelihood-ratio test as a model test of the linear logistic test model. Journal of Applied Measurement, 15, 252–266.
   PubMedGoogle Scholar
• Holzinger, K. J., & Swineford, F. (1939). A study in factor analysis: The stability of a bi-factor solution. Chicago, IL: The University of Chicago.
   Google Scholar
• Howard, G. S., Lau, M. Y., Maxwell, S. E., Venter, A., Lundy, R., & Sweeny, R. M. (2009). Do research literatures give correct answers? Review of General Psychology, 13, 116–121. doi:10.1037/a0015468
   Web of Science ®Google Scholar
• Hurvich, C., & Tsai, C.-L. (1990). The impact of model selection on inference in linear regression. The American Statistician, 44, 214.
   Web of Science ®Google Scholar
• Jiao, Y., Neves, R., & Jones, J. (2008). Models and model selection uncertainty in estimating growth rates of endangered freshwater mussel populations. Canadian Journal of Fisheries and Aquatic Sciences, 65, 2389–2398. doi:10.1139/F08-141
   Web of Science ®Google Scholar
• Lubke, G. H. (2012). Old issues in a new jacket: Power and validation in the context of mixture modeling. Measurement, 10, 212–216.
   Google Scholar
• Lubke, G. H., & Muthén, B. (2005). Investigating population heterogeneity with factor mixture models. Psychological Methods, 10, 21–39. doi:10.1037/1082-989X.10.1.21
   PubMed Web of Science ®Google Scholar
• Lubke, G. H., & Spies, J. (2008). Choosing a “correct” factor mixture model: Power, limitations, and graphical data exploration. In G. R. Hancock & K. M. Samuelsen (Eds.), Advances in latent variable mixture models (pp. 343–362). Charlotte, NC: Information Age.
   Google Scholar
• MacCallum, R. C., Browne, M. W., & Sugawara, H. M. (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1, 130–149. doi:10.1037/1082-989X.1.2.130
   Web of Science ®Google Scholar
• Marcoulides, G. A. (1989). The estimation of variance components of in generalizability studies: A resampling approach. Psychological Reports, 65, 883–889. doi:10.2466/pr0.1989.65.3.883
   Web of Science ®Google Scholar
• Marcoulides, G. A. (1990). Evaluation of confirmatory factor analytic and structural equation models using goodness-of-fit indices. Psychological Reports, 67, 669–670. doi:10.2466/pr0.1990.67.2.669
   Web of Science ®Google Scholar
• Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58, 525–543. doi:10.1007/BF02294825
   Web of Science ®Google Scholar
• Merkle, E. C., You, D., & Preacher, K. J. (2015). Testing nonnested structural equation models. Psychological Methods. Advance online publication. doi:10.1037/met0000038
   PubMedGoogle Scholar
• Muthén, B., & Muthén, L. K. (2000). Integrating person-centered and variable-centered analyses: Growth mixture modeling with latent trajectory classes. Alcoholism, Clinical and Experimental Research, 24, 882–891. doi:10.1111/j.1530-0277.2000.tb02070.x
   PubMed Web of Science ®Google Scholar
• Muthén, B., & Shedden, K. (1999). Finite mixture modeling with mixture outcomes using the EM algorithm. Biometrics, 55, 463–469. doi:10.1111/j.0006-341X.1999.00463.x
   PubMed Web of Science ®Google Scholar
• Muthén, L. K., & Muthén, B. O. (1998–2015). Mplus user’s guide (7th ed.). Los Angeles, CA: Muthén & Muthén.
   Google Scholar
• Nagin, D. (2005). Group-based modeling of development. Cambridge, MA: Harvard University Press.
   Google Scholar
• Nagin, D. S., & Land, K. C. (1993). Age, criminal careers, and population heterogeneity: Specification and estimation of a nonparametric, mixed Poisson model. Criminology, 31, 327–362. doi:10.1111/crim.1993.31.issue-3
   Web of Science ®Google Scholar
• Nylund, K., Asparouhov, T., & Muthén, B. (2007). Deciding on the number of classes in latent class analysis and growth mixture modeling: A Monte Carlo simulation study. Structural Equation Modeling, 14, 535–569. doi:10.1080/10705510701575396
   Web of Science ®Google Scholar
• Parker, C., & Hawkins, N. (2015). The use of bootstrap model averaging when estimating survival curves. Value in Health, 18(3), A23. doi:10.1016/j.jval.2015.03.144
   Google Scholar
• Preacher, K. J., & Merkle, E. C. (2012). The problem of model selection uncertainty in structural equation modeling. Psychological Methods, 17, 1–14. doi:10.1037/a0026804
   PubMed Web of Science ®Google Scholar
• Roberts, S., & Martin, M. A. (2010). Does ignoring model selection when assessing the effect of particulate matter air pollution on mortality make us too vigilant? Annals of Epidemiology, 20, 772–778. doi:10.1016/j.annepidem.2010.03.019
   PubMed Web of Science ®Google Scholar
• Satorra, A., Saris, W. E., & Pijper, W. M. (1991). A comparison of several approximations to the power function of the likelihood ratio test in covariance structure analysis. Statistica Neerlandica, 45, 173–185. doi:10.1111/j.1467-9574.1991.tb01302.x
   Google Scholar
• Shao, J. (1996). Bootstrap model selection. Journal of the American Statistical Association, 91, 655–665. doi:10.1080/01621459.1996.10476934
   Web of Science ®Google Scholar
• Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology. Psychological Science, 22, 1359–1366. doi:10.1177/0956797611417632
   PubMed Web of Science ®Google Scholar
• Simonoff, E. (2000). Extracting meaning from comorbidity: Genetic analyses that make sense. Journal of Child Psychology and Psychiatry, 41, 667–674. doi:10.1111/1469-7610.00653
   PubMed Web of Science ®Google Scholar
• Tukey, J. W. (1980). Methodological comments focused on opportunities. Multivariate Techniques in Human Communication Research, 490–528.
   Google Scholar
• Wang, H.-F. (2010). The impact of preliminary model selection on latent growth model parameter estimates ( Unpublished doctoral dissertation). Department of Measurement, Statistics, and Evaluation, University of Maryland, College Park, MD.
   Google Scholar
• Whittaker, T. A., & Stapleton, L. M. (2006). The performance of cross-validation indices used to select among competing covariance structure models under multivariate nonnormality conditions. Multivariate Behavioral Research, 41, 295–335. doi:10.1207/s15327906mbr4103_3
   PubMed Web of Science ®Google Scholar
• Zhang, P. (1992). Inference after variable selection in linear regression models. Biometrika, 79, 741–746. doi:10.1093/biomet/79.4.741
   Web of Science ®Google Scholar