Robust sparse regression by modeling noise as a mixture of gaussians

ABSTRACT

Regression analysis has been proven to be a quite effective tool in a large variety of fields. In many regression models, it is often assumed that noise is with a specific distribution. Although the theoretical analysis can be greatly facilitated, the model-fitting performance may be poor since the supposed noise distribution may deviate from real noise to a large extent. Meanwhile, the model is also expected to be robust in consideration of the complexity of real-world data. Without any assumption about noise, we propose in this paper a novel sparse regression method called MoG-Lasso to directly model noise in linear regression models via a mixture of Gaussian distributions (MoG). Meanwhile, the $L_1$ penalty is included as a part of the loss function of MoG-Lasso to enhance its ability to identify a sparse model. As for the parameters in MoG-Lasso, we present an efficient algorithm to estimate them via the EM (expectation maximization) and ADMM (alternating direction method of multipliers) algorithms. With some simulated and real data contaminated by complex noise, the experiments show that the novel model MoG-Lasso performs better than several other popular methods in both ‘p>n’ and ‘p<n’ situations, including Lasso, LAD-Lasso and Huber-Lasso.

KEYWORDS:

• Robust regression
• penalized regression
• variable selection
• mixture of Gaussians
• lasso

Acknowledgements

The authors would like to thank the editor and the referees for their valuable comments which greatly helped to improve the paper.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. https://CRAN.R-project.org/package=glmnet

2. https://CRAN.R-project.org/package=hqreg

3. http://www.dcc.fc.up.pt/∼ltorgo/Regression/DataSets.html

4. https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE5680

Additional information

Funding

This work was supported by the National Natural Science Foundation of China [grant number 11671317].