ENTROPY MINIMAX MULTIVARIATE STATISTICAL MODELING–I: THEORY

Abstract

The theory of entropy mimmax, an information theoretic approach to predictive modeling, is reviewed. Comparisons are given to other methodologies, including: Neyman-Pearson hypothesis testing, James-Stein “empirical Bayes” estimation, maximum likelihood estimation, least-squares fitting, linear regression and logistic regression. Examples are provided showing how maximum entropy expectation probabilities are computed and how minimum entropy partitions are determined. The importance of the a priori weight normalization, in establishing the coarse-grain of the minimum entropy partition, is discussed. The trial crossvalidation procedure for determining the normalization is described. Generalizations utilizing Zadeh's fuzzy entropies are provided for variables involving indistinguishability, partial or total. Specific cases are discussed of maximum likelihood estimation, illustrating ils “data range irregularity” which is avoided by methods such as entropy minimization that account for residuals over the full distribution range. Discriminant analysis and polynomial fitting are discussed as examples of areas of application of the principles of entropy minimax. In curve fitting, the order of the polynomial is determined by entropy minimization, and the coefficients are determined by entropy maximization. Entropy maximization operates as a goodness-of-fit criterion, while entropy minimization operates as a mathematical formulation of Ockham's razor, controlling the level of smoothness in fitting models to data.

INDEX TERMS:

• Bayesian statistics
• curve fitting
• discriminant analysis
• empirical Bayes
• entropy
• entropy minimax
• experimental design
• fuzzy sets
• hypothesis testing
• induction
• information
• maximum entropy
• maximum likelihood
• minimum entropy
• multivariate statistics
• Ockham's razor
• probability
• smoothing
• statistical modeling
• statistical prediction
• systems modeling.