Sparse Reduced Rank Huber Regression in High Dimensions

Abstract

We propose a sparse reduced rank Huber regression for analyzing large and complex high-dimensional data with heavy-tailed random noise. The proposed method is based on a convex relaxation of a rank- and sparsity-constrained nonconvex optimization problem, which is then solved using a block coordinate descent and an alternating direction method of multipliers algorithm. We establish nonasymptotic estimation error bounds under both Frobenius and nuclear norms in the high-dimensional setting. This is a major contribution over existing results in reduced rank regression, which mainly focus on rank selection and prediction consistency. Our theoretical results quantify the tradeoff between heavy-tailedness of the random noise and statistical bias. For random noise with bounded $(1+\delta )$th moment with $\delta \in (0,1)$, the rate of convergence is a function of δ, and is slower than the sub-Gaussian-type deviation bounds; for random noise with bounded second moment, we obtain a rate of convergence as if sub-Gaussian noise were assumed. We illustrate the performance of the proposed method via extensive numerical studies and a data application. Supplementary materials for this article are available online.

KEYWORDS:

• Convex relaxation
• Huber loss
• Low rank approximation
• Sparsity

Supplementary Materials

The online supplementary materials collect an ADMM algorithm, the proofs for all the theoretical results.

Notes

1 We note that the prediction error bound can be used to derive an estimation error bound under some further incoherence condition on the design matrix.

Additional information

Funding

Tan is supported by NSF DMS 2113356, NSF DMS 1949730, and NIH RF1-MH122833. Sun is supported in part by NSERC grant RGPIN-2018-06484. Witten is supported by NIH R01 GM123993, Simons Investigator for Mathematical Modeling of Living Systems and NSF CAREER award 1252624.