Regularized Gaussian Discriminant Analysis through Eigenvalue Decomposition

Abstract

Friedman proposed a regularization technique (RDA) of discriminant analysis in the Gaussian framework. RDA uses two regularization parameters to design an intermediate classifier between the linear, the quadratic the nearest-means classifiers. In this article we propose an alternative approach, called EDDA, that is based on the reparameterization of the covariance matrix [Σ _k ] of a group G_k in terms of its eigenvalue decomposition Σ _k = λ _k D _k A _k D _k ′, where λk specifies the volume of density contours of Gk, the diagonal matrix of eigenvalues specifies its shape the eigenvectors specify its orientation. Variations on constraints concerning volumes, shapes orientations λ _k , A _k , and D _k lead to 14 discrimination models of interest. For each model, we derived the normal theory maximum likelihood parameter estimates. Our approach consists of selecting a model by minimizing the sample-based estimate of future misclassification risk by cross-validation. Numerical experiments on simulated and real data show favorable behavior of this approach compared to RDA.

Key Words:

• Covariance matrix
• Maximum likelihood
• Normal-based classification
• Spectral decomposition