Redundancy reduction with information-preserving nonlinear maps

Abstract

The basic idea of linear principal component analysis (PCA) involves decorrelating coordinates by an orthogonal linear transformation. In this paper we generalize this idea to the nonlinear case. Simultaneously we shall drop the usual restriction to Gaussian distributions. The linearity and orthogonality condition of linear PCA is replaced by the condition of volume conservation in order to avoid spurious information generated by the nonlinear transformation. This leads us to another very general class of nonlinear transformations, called symplectic maps. Later, instead of minimizing the correlation, we minimize the redundancy measured at the output coordinates. This generalizes second-order statistics, being only valid for Gaussian output distributions, to higher-order statistics. The proposed paradigm implements Barlow's redundancy-reduction principle for unsupervised feature extraction. The resulting factorial representation of the joint probability distribution presumably facilitates density estimation and is applied in particular to novelty detection.