Information-Theoretic Distribution Test with Application to Normality

Abstract

We derive general distribution tests based on the method of maximum entropy (ME) density. The proposed tests are derived from maximizing the differential entropy subject to given moment constraints. By exploiting the equivalence between the ME and maximum likelihood (ML) estimates for the general exponential family, we can use the conventional likelihood ratio (LR), Wald, and Lagrange multiplier (LM) testing principles in the maximum entropy framework. In particular, we use the LM approach to derive tests for normality. Monte Carlo evidence suggests that the proposed tests are compatible with and sometimes outperform some commonly used normality tests. We show that the proposed tests can be extended to tests based on regression residuals and non-i.i.d. data in a straightforward manner. An empirical example on production function estimation is presented.

Keywords:

• Distribution test
• Maximum entropy
• Normality

JEL Classification:

• C1
• C12
• C16

ACKNOWLEDGMENTS

We want to thank the associate editor, two anonymous referees, seminar participants at Penn State University, the 2004 European Meeting of the Econometric Society, and the 2004 Canadian Econometrics Study Group for comments. Financial support from SSHRC of Canada is gratefully acknowledged.

Notes

Let γ′ = [γ₀, γ₁,…, γ _K ] be a nonzero vector and g ₀(x) = 1, we have

Hence, ℋ is positive-definite.

Denote θ _m  = [θ₁,…, θ _m ]. The only case that θ _m+1 = 0 is when the moment restriction is not binding, or the (m + 1)th moment is identical to its prediction based on the ME density f(x; θ _m ) from the first m moments. In this case, the (m + 1)th moment contains no additional information that can further reduce the entropy.

Imbens et al. (1998) discussed similar tests in the IT generalized empirical likelihood framework. The proposed tests differ from their tests, which minimize the discrete Kullback–Leibler information criterion (cross entropy) or other Cressie–Read family of discrepancy indices subject to moment constraints.

A t distribution with one degree of freedom is the Cauchy distribution, which has the fattest tails within the family of t distributions. See also Lye and Martin (1994) on the connection between testing for normality and the generalized Student t distribution. Premaratne and Bera (2005) also used the moment function tan ⁻¹(x).

We also tried [tan ⁻¹(x)]². The performance was essentially the same as that with tan ⁻¹(x ²).

Shannon (1949) shows that among all distributions that possess a density function f(x) and have a given variance σ², the entropy is maximized by the normal distribution. The entropy of the normal distribution with variance σ² is . Vasicek (1976) uses this property to test a composite hypothesis of normality, based on a nonparametric estimates of sample entropy.

The first two moments are zero and one by standardization.

We thank a referee for this reference.