Regularization with Maximum Entropy and Quantum Electrodynamics: The Merg(E) Estimators

Abstract

It is well-known that under fairly conditions linear regression becomes a powerful statistical tool. In practice, however, some of these conditions are usually not satisfied and regression models become ill-posed, implying that the application of traditional estimation methods may lead to non-unique or highly unstable solutions. Addressing this issue, in this paper a new class of maximum entropy estimators suitable for dealing with ill-posed models, namely for the estimation of regression models with small samples sizes affected by collinearity and outliers, is introduced. The performance of the new estimators is illustrated through several simulation studies.

Keywords:

• collinearity
• linear regression
• maximum entropy
• micronumerosity
• outliers
• quantum electrodynamics

Mathematics Subject Classification::

• 47A52
• 62J05
• 62J07
• 81V10
• 94A17

Notes

¹The weights at each iteration are calculated by applying Tukey’s biweight function to the residuals from the previous iteration.

²Other ridge and Liu-type estimators are also considered, but the results (not reported here) are very poor. These estimators depend on some parameters that must be estimated from the sample and the results are sensitive to the quality of these parameter estimates.

³All the MSEL values presented for the RR-MM estimator (in Table 2 and others where it is used) are calculated using a 10% upper trimmed average; see Maronna (2011, p. 49). The real values of MSEL are higher.

⁴Results from different empirical applications are not shown here due to space limitations, but are provided upon request to the authors.

⁵The idea of a weighted GME objective function, which is followed here for the MERGE estimators, is proposed in Wu (2009).

⁶The exogenous parameter weighting between signal and noise is θ = 0.5 in this study. Naturally, this parameter can be changed in order to reflect different weights in the components of the objective function. The impact of this choice is left for future research.

⁷The MSEL values for the RR-MM estimator are calculated using a 10% upper trimmed average.