Striving for Sparsity: On Exact and Approximate Solutions in Regularized Structural Equation Models

References

• Andrew, G., & Gao, J. (2007). Scalable training of L1-regularized log-linear models. In Proceedings of the 24th International Conference on Machine Learning – ICML ’07 (pp. 33–40). ACM Press.
   Google Scholar
• Battauz, M. (2020). Regularized estimation of the nominal response model. Multivariate Behavioral Research, 55, 811–824. https://doi.org/10.1080/00273171.2019.1681252
   PubMed Web of Science ®Google Scholar
• Belzak, W. C. M. (2021). Using regularization to evaluate differential item functioning among multiple covariates: A penalized expectation-maximization algorithm via coordinate descent and soft-thresholding [doctoral dissertation]. University of North Carolina, Chapel Hill.
   Google Scholar
• Belzak, W. C. M., & Bauer, D. J. (2020). Improving the assessment of measurement invariance: Using regularization to select anchor items and identify differential item functioning. Psychological Methods, 25, 673–690. https://doi.org/10.1037/met0000253
   PubMed Web of Science ®Google Scholar
• Brandmaier, A. M., & Jacobucci, R. C. (in press). Machine-learning approaches to structural equation modeling. In R. H. Hoyle (Ed.), Handbook of structural equation modeling (2nd ed.). Guilford Press.
   Google Scholar
• Breheny, P., & Huang, J. (2011). Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection. The Annals of Applied Statistics, 5, 232–253. https://doi.org/10.1214/10-AOAS388
   PubMed Web of Science ®Google Scholar
• Browne, M. W. (2000). Cross-validation methods. Journal of Mathematical Psychology, 44, 108–132. https://doi.org/10.1006/jmps.1999.1279
   PubMed Web of Science ®Google Scholar
• Candès, E. J., Wakin, M. B., & Boyd, S. P. (2008). Enhancing sparsity by reweighted l1 minimization. Journal of Fourier Analysis and Applications, 14, 877–905. https://doi.org/10.1007/s00041-008-9045-x
   Web of Science ®Google Scholar
• Cao, Y., Huang, J., Jiao, Y., & Liu, Y. (2017). A lower bound based smoothed quasi-Newton algorithm for group bridge penalized regression. Communications in Statistics-Simulation and Computation, 46, 4694–4707. https://doi.org/10.1080/03610918.2015.1129409
   Web of Science ®Google Scholar
• Chamroukhi, F., & Huynh, B.-T. (2019). Regularized maximum likelihood estimation and feature selection in mixtures-of-experts models. Journal de la Société Française de Statistique, 160, 57–85.
   Google Scholar
• Davis, S. W., Szymanski, A., Boms, H., Fink, T., & Cabeza, R. (2019). Cooperative contributions of structural and functional connectivity to successful memory in aging. Network Neuroscience, 3, 173–194. https://doi.org/10.1162/netn_a_00064
   PubMed Web of Science ®Google Scholar
• Eddelbuettel, D., & François, R. (2011). Rcpp: Seamless R and C++ integration. Journal of Statistical Software, 40, 1–18. https://doi.org/10.18637/jss.v040.i08
   Web of Science ®Google Scholar
• Eddelbuettel, D., & Sanderson, C. (2014). RcppArmadillo: Accelerating R with high-performance C++ linear algebra. Computational Statistics & Data Analysis, 71, 1054–1063. https://doi.org/10.1016/j.csda.2013.02.005
   Web of Science ®Google Scholar
• Epskamp, S., Rhemtulla, M., & Borsboom, D. (2017). Generalized network psychometrics: Combining network and latent variable models. Psychometrika, 82, 904–927. https://doi.org/10.1007/s11336-017-9557-x
   PubMed Web of Science ®Google Scholar
• Fan, J., & Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association, 96, 1348–1360. https://doi.org/10.1198/016214501753382273
   Web of Science ®Google Scholar
• Fan, R.-E., Chang, K.-W., Hsieh, C.-J., Wang, X.-R., & Lin, C.-J. (2008). LIBLINEAR: A library for large linear classification. Journal of Machine Learning Research, 9, 1871–1874.
   Web of Science ®Google Scholar
• Finch, W. H., & Miller, J. E. (2021). A comparison of regularized maximum-likelihood, regularized 2-stage least squares, and maximum-likelihood estimation with misspecified models, small samples, and weak factor structure. Multivariate Behavioral Research, 56, 608–626. https://doi.org/10.1080/00273171.2020.1753005
   PubMed Web of Science ®Google Scholar
• Friedman, J., Hastie, T., & Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33, 1–20. https://doi.org/10.18637/jss.v033.i01
   PubMed Web of Science ®Google Scholar
• Friemelt, B., Bloszies, C., Ernst, M. S., Peikert, A., Brandmaier, A. M., & Koch, T. (2022). On the performance of different regularization methods in bifactor-(S-1) models with explanatory variables—Caveats, recommendations, and future directions. Structural Equation Modeling: A Multidisciplinary Journal. Advance online publication. https://doi.org/10.1080/10705511.2022.2140664
   Web of Science ®Google Scholar
• Geminiani, E., Marra, G., & Moustaki, I. (2021). Single- and multiple-group penalized factor analysis: A trust-region algorithm approach with integrated automatic multiple tuning parameter selection. Psychometrika, 86, 65–95. https://doi.org/10.1007/s11336-021-09751-8
   PubMed Web of Science ®Google Scholar
• Ghalanos, A., & Theussl, S. (2015). Rsolnp: General non-linear optimization using augmented lagrange multiplier method.
   Google Scholar
• Goldstein, T., Studer, C., & Baraniuk, R. (2016). A field guide to forward-backward splitting with a FASTA implementation (No. arXiv:1411.3406). arXiv.
   Google Scholar
• Gong, P., Zhang, C., Lu, Z., Huang, J., Ye, J. (2013). A general iterative shrinkage and thresholding algorithm for non-convex regularized optimization problems. In Proceedings of the 30th International Conference on Machine Learning (Vol. 28, pp. 37–45). PMLR.
   Google Scholar
• Góngora, D., Vega-Hernández, M., Jahanshahi, M., Valdés-Sosa, P. A., Bringas-Vega, M. L., & C. (2020). Crystallized and fluid intelligence are predicted by microstructure of specific white-matter tracts. Human Brain Mapping, 41, 906–916. https://doi.org/10.1002/hbm.24848
   PubMed Web of Science ®Google Scholar
• Hastie, T., Tibshirani, R., & Wainwright, M. (2015). Statistical learning with sparsity: The lasso and generalizations. CRC Press.
   Google Scholar
• Hoerl, A. E., & Kennard, R. W. (1970). Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12, 55–67. https://doi.org/10.1080/00401706.1970.10488634
   Web of Science ®Google Scholar
• Huang, P.-H. (2018). A penalized likelihood method for multi-group structural equation modelling. British Journal of Mathematical and Statistical Psychology, 71, 499–522. https://doi.org/10.1111/bmsp.12130
   PubMed Web of Science ®Google Scholar
• Huang, P.-H. (2020a). Lslx: Semi-confirmatory structural equation modeling via penalized likelihood. Journal of Statistical Software, 93, 1–37. https://doi.org/10.18637/jss.v093.i07
   Web of Science ®Google Scholar
• Huang, P.-H. (2020b). Penalized least squares for structural equation modeling with ordinal responses. Multivariate Behavioral Research, 57, 279–297. https://doi.org/10.1080/00273171.2020.1820309
   PubMed Web of Science ®Google Scholar
• Huang, P.-H., Chen, H., & Weng, L.-J. (2017). A penalized likelihood method for structural equation modeling. Psychometrika, 82, 329–354. https://doi.org/10.1007/s11336-017-9566-9
   PubMed Web of Science ®Google Scholar
• Jacobucci, R. (2017). Regsem: Regularized structural equation modeling. arXiv:1703.08489 [stat].
   Google Scholar
• Jacobucci, R., Brandmaier, A. M., & Kievit, R. A. (2019). A practical guide to variable selection in structural equation modeling by using regularized multiple-indicators, multiple-causes models. Advances in Methods and Practices in Psychological Science, 2, 55–76. https://doi.org/10.1177/2515245919826527
   PubMedGoogle Scholar
• Jacobucci, R., & Grimm, K. J. (2018). Regularized estimation of multivariate latent change score models. In E. Ferrer, S. M. Boker, & K. J. Grimm (Eds.), Longitudinal multivariate psychology (1st ed., pp. 109–125). Routledge.
   Google Scholar
• Jacobucci, R., Grimm, K. J., & McArdle, J. J. (2016). Regularized structural equation modeling. Structural Equation Modeling: A Multidisciplinary Journal, 23, 555–566. https://doi.org/10.1080/10705511.2016.1154793
   PubMed Web of Science ®Google Scholar
• James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning: With applications in R (No. 103). Springer.
   Google Scholar
• Kay, M. (2022). {ggdist}: Visualizations of distributions and uncertainty.
   Google Scholar
• Koncz, R., Wen, W., Makkar, S. R., Lam, B. C. P., Crawford, J. D., Rowe, C. C., & Sachdev, P. (2022). The interaction between vascular risk factors, cerebral small vessel disease, and amyloid burden in older adults. Journal of Alzheimer’s Disease, 86, 1617–1628. https://doi.org/10.3233/JAD-210358
   PubMed Web of Science ®Google Scholar
• Konishi, S., & Kitagawa, G. (1996). Generalised information criteria in model selection. Biometrika, 83, 875–890. https://doi.org/10.1093/biomet/83.4.875
   Web of Science ®Google Scholar
• Lee, S.-I., Lee, H., Abbeel, P., Ng, A. Y. (2006). Efficient l1 regularized logistic regression. In Proceedings of the Twenty-First National Conference on Artificial Intelligence (AAAI-06) (pp. 401–408).
   Google Scholar
• Li, X., & Jacobucci, R. (2021). Regularized structural equation modeling with stability selection. Psychological Methods, 27, 497–518. https://doi.org/10.1037/met0000389
   PubMed Web of Science ®Google Scholar
• Li, X., Jacobucci, R., & Ammerman, B. A. (2021). Tutorial on the use of the regsem package in R. Psych, 3, 579–592. https://doi.org/10.3390/psych3040038
   Google Scholar
• Marsh, H. W., Lüdtke, O., Muthén, B., Asparouhov, T., Morin, A. J. S., Trautwein, U., et al. (2010). A new look at the big five factor structure through exploratory structural equation modeling. Psychological Assessment, 22, 471–491. https://doi.org/10.1037/a0019227
   PubMed Web of Science ®Google Scholar
• Muggeo, V. M. R. (2010). LASSO regression via smooth L1-norm approximation. In Proceedings of the 25th International Workshop on Statistical Modelling (pp. 391–396). Glasgow.
   Google Scholar
• Neale, M. C., Hunter, M. D., Pritikin, J. N., Zahery, M., Brick, T. R., Kirkpatrick, R. M., et al. (2016). OpenMx 2.0: Extended structural equation and statistical modeling. Psychometrika, 81, 535–549. https://doi.org/10.1007/s11336-014-9435-8
   PubMed Web of Science ®Google Scholar
• Nocedal, J., & Wright, S. J. (2006). Numerical optimization (2nd ed.). Springer.
   Google Scholar
• Orzek, J. H., & Voelkle, M. C. (2023). Regularized continuous time structural equation models: A network perspective. Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000550
   PubMed Web of Science ®Google Scholar
• Parikh, N., & Boyd, S. (2013). Proximal algorithms. Foundations and Trends in Optimization, 1, 123–231.
   Google Scholar
• Petersen, K. B., & Pedersen, M. S. (2012). Matrix cookbook. Technical University of Denmark.
   Google Scholar
• R Core Team (2022). R: A language and environment for statistical computing. R Foundation for Statistical Computing.
   Google Scholar
• Revelle, W. (2022). psychTools: Tools to accompany the psych package for psychological research. Northwestern University.
   Google Scholar
• Revelle, W., Wilt, J., & Rosenthal, A. (2010). Individual differences in cognition: New methods for examining the personality-cognition link. In A. Gruszka, G. Matthews, & B. Szymura (Eds.), Handbook of individual differences in cognition: Attention, memory, and executive control (pp. 27–49). Springer.
   Google Scholar
• Ridler, I. (2022). Applications of regularised structural equation modelling to psychometrics and behavioural genetics [doctoral dissertation]. King’s College.
   Google Scholar
• Rosseel, Y. (2012). Lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48, 1–36. https://doi.org/10.18637/jss.v048.i02
   Web of Science ®Google Scholar
• Scharf, F., & Nestler, S. (2019). Should regularization replace simple structure rotation in exploratory factor analysis? Structural Equation Modeling: A Multidisciplinary Journal, 26, 576–590. https://doi.org/10.1080/10705511.2018.1558060
   Web of Science ®Google Scholar
• Schmidt, M., Fung, G., & Rosaless, R. (2009). Optimization methods for l1 regularization. Technical Report TR-2009-19. University of British Columbia.
   Google Scholar
• Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6, 461–464. https://doi.org/10.1214/aos/1176344136
   Web of Science ®Google Scholar
• Shao, J. (1997). An asymptotic theory for linear model selection. Statistica Sinica, 7, 221–242.
   Web of Science ®Google Scholar
• Stone, M. (1977). An asymptotic equivalence of choice of model by cross-validation and Akaike’s criterion. Journal of the Royal Statistical Society: Series B (Methodological), 39, 44–47. https://doi.org/10.1111/j.2517-6161.1977.tb01603.x
   Web of Science ®Google Scholar
• Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological), 58, 267–288.
   Web of Science ®Google Scholar
• Ulbricht, J. (2010). Variable selection in generalized linear models [doctoral dissertation]. Ludwig-Maximilians-Universität München.
   Google Scholar
• Voelkle, M. C., Oud, J. H. L., Davidov, E., & Schmidt, P. (2012). An SEM approach to continuous time modeling of panel data: Relating authoritarianism and anomia. Psychological Methods, 17, 176–192. https://doi.org/10.1037/a0027543
   PubMed Web of Science ®Google Scholar
• von Oertzen, T., & Brick, T. R. (2013). Efficient Hessian computation using sparse matrix derivatives in RAM notation. Behavior Research Methods, 46, 385–395. https://doi.org/10.3758/s13428-013-0384-4
   Web of Science ®Google Scholar
• Wickham, H. (2016). Ggplot2: Elegant graphics for data analysis (2nd ed.). Springer International Publishing.
   Google Scholar
• Ye, A., Gates, K. M., Henry, T. R., & Luo, L. (2021). Path and directionality discovery in individual dynamic models: A regularized unified structural equation modeling approach for hybrid vector autoregression. Psychometrika, 86, 404–441.
   PubMed Web of Science ®Google Scholar
• Ye, Y. (1987). Interior algorithms for linear, quadratic, and linearly constrained non-linear programming [doctoral dissertation]. Department of EES, Stanford University.
   Google Scholar
• Yuan, G.-X., Chang, K.-W., Hsieh, C.-J., & Lin, C.-J. (2010). A comparison of optimization methods and software for large-scale l1-regularized linear classification. Journal of Machine Learning Research, 11, 3183–3234.
   Web of Science ®Google Scholar
• Yuan, G.-X., Ho, C.-H., & Lin, C.-J. (2012). An improved GLMNET for l1-regularized logistic regression. The Journal of Machine Learning Research, 13, 1999–2030. https://doi.org/10.1145/2020408.2020421
   Web of Science ®Google Scholar
• Zhang, C.-H. (2010). Nearly unbiased variable selection under minimax concave penalty. The Annals of Statistics, 38, 894–942. https://doi.org/10.1214/09-AOS729
   Web of Science ®Google Scholar
• Zhang, T. (2010). Analysis of multi-stage convex relaxation for sparse regularization. Journal of Machine Learning Research, 11, 1081–1107.
   Web of Science ®Google Scholar
• Zou, H. (2006). The adaptive lasso and its oracle properties. Journal of the American Statistical Association, 101, 1418–1429. https://doi.org/10.1198/016214506000000735
   Web of Science ®Google Scholar
• Zou, H., & Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B, 67, 301–320. https://doi.org/10.1111/j.1467-9868.2005.00503.x
   Google Scholar
• Zou, H., & Li, R. (2008). One-step sparse estimates in nonconcave penalized likelihood models. Annals of Statistics, 36, 1509–1533. https://doi.org/10.1214/009053607000000802
   PubMed Web of Science ®Google Scholar