Striving for Sparsity: On Exact and Approximate Solutions in Regularized Structural Equation Models

Abstract

Regularized structural equation models have gained considerable traction in the social sciences. They promise to reduce overfitting by focusing on out-of-sample predictions and sparsity. To this end, a set of increasingly constrained models is fitted to the data. Subsequently, one of the models is selected, usually by means of information criteria. Current implementations of regularized structural equation models differ in their optimizers: Some use general purpose optimizers whereas others use specialized optimization routines. While both approaches often perform similarly, we show that they can produce very different results. We argue that in particular, the interaction between optimizer and selection criterion (e.g., BIC) contributes to these differences. We substantiate our arguments with an empirical demonstration and a simulation study. Based on these findings, we conclude that researchers should consider specialized optimizers whenever possible. To facilitate the implementation of such optimizers, we provide the R package lessSEM.

Keywords:

• Lasso
• optimization
• regularization
• structural equation model

Acknowledgments

During the work on his dissertation, Jannik H. Orzek was a pre-doctoral fellow of the International Max Planck Research School on the Life Course (https://www.imprs-life.mpg.de/) of the Max Planck Institute for Human Development, Berlin, Germany. We acknowledge support by the Open Access Publication Fund of Humboldt-Universität zu Berlin. Figures in this article were created with ggplot2 (Wickham, 2016) and ggdist (Kay, 2022). The online supplement can be found at https://osf.io/kh9tr/.

Notes

1 For instance, lslx offers multi-group regularization, whereas regsem allows for equality constraints of parameters.

2 The gradient is a vector of first derivatives of the fitting function with respect to the parameters and the Hessian is a matrix with second derivatives.

3 The threshold parameter τ can be adapted with the round parameter in regsem; see regsem-function in the regsem package.

4 The lessSEM (lessSEM estimates sparse SEM) version used in this study is available from https://osf.io/kh9tr/. The most recent version can be found at https://github.com/jhorzek/lessSEM. Similar to regsem, lessSEM builds on models from lavaan (Rosseel, 2012).

5 We compare our implementation with lslx in the osf repository at https://osf.io/kh9tr/.

6 See fasta (Goldstein et al., 2016) for an alternative R package implementing general purpose optimization of non-differentiable penalty functions in pure R.

7 Files to replicate the simulations and the empirical example are provided in the osf repository at https://osf.io/kh9tr/. All analyses were performed using a single core of an Intel^® Xeon^® E5-2670 processor.

8 We chose this way of scaling because scaling by setting the variances to 1 would allow for multiple equivalent solutions with reversed signs of loadings.

9 In regsem $\tau ={10}^{-3}$ is the current default, while Belzak and Bauer (2020) and Epskamp et al. (2017) used $\tau ={10}^{-5}.$

10 The best BIC, for example, would always be achieved by setting $\tau =\infty$ and λ = 0. Then, the BIC reduces to $F_{\text{ML}}({\widehat{\mathit{\theta }}}_{\text{ML}})$ (the −2 log-likelihood of the maximum likelihood parameter estimates) plus $\hspace{0.17em}\log \hspace{0.17em}(N)$ times the number of unregularized parameters.