Abstract
While consequences of unobserved heterogeneity such as biased estimates of binary response regression models are generally known; quantifying these and awareness of situations with more serious impact on inference is however, remarkably lacking. This study examines the effect of unobserved heterogeneity on estimates of the standard logistic model. An estimate of bias was derived for the maximum likelihood estimator , and simulated data was used to investigate a range of situations that influence size of bias due to unobserved heterogeneity. It was found that the position of the probabilities, along the logistic curve, and the variance of the unobserved heterogeneity, were important determinants of size of bias.
Mathematics Subject Classification:
1. Introduction
Theoretical models, such as health modes, generally conceptualize outcomes as a result of interaction among a complex set of components including biological, genetic, behavioral, and socio-economics (Mosley and Chen, Citation2003; World Health Organization Citation2001). In practical situations of data analysis, however, it is not possible to account for all variables that result in an outcome by including these as explanatory variables in a statistical model. Even in the richest model specification, several factors would be unobserved, immeasurable or unknown, and some of these would be of high importance to the resulting outcome (Lee and Lee, Citation2003; Zohoori and Savitz Citation1997). Nonetheless, it is not uncommon in many specialized journals to find some conclusions that were reached on the basis of inference from observed variables, assuming unobserved heterogeneity is of little relevance. Economists were puzzled for nearly two decades by the spurious positive association between drinking alcohol (medically known as drug with depressant properties and is unlikely to positively affect productivity) and high wage, where drinkers were persistently found to earn more than alcohol abstainers, the association was reversed by the introduction of individual specific fixed effect, which rid results of bias due to time invariant unobserved heterogeneity (Peters, Citation2004).
Unobserved factors whether environmental or personal may have a large effect on outcomes of relevance (for example, health, economics, social). Ignoring such effects may lead to the identification of incorrect risk factors, in addition the magnitude of the association of these may be so seriously biased and as a result conclusions may be misleading.
The objectives of this study were to quantify bias of the maximum likelihood (ML) estimate, of a standard logistic model due to un-modeled unobserved heterogeneity, and to highlight situations where the impact of such bias was more serious. Using Taylor series theory, an approximate estimate of bias was derived then simulation was used to investigate situations that were thought to affect the size of bias. The importance of higher-order terms of the derived approximation was also examined under various situations, including different variance of unobserved heterogeneity and differing positions for the probabilities of outcome within the logistic curve. The estimation described was confined to the case of a single binary explanatory variable x, a binary response y, and unobserved heterogeneity that was linked to each individual.
It was found that in most situations the first-order approximation defined as δ1, provides an adequate approximation of bias due to unobserved heterogeneity. At special situations with large variance and large difference between the two probabilities, however, the first-order approximation becomes inadequate, and does worse as the difference between the two probabilities increases.
2. Methods
2.1. Assumptions and Models
We consider a hypothetical simple example, i.e., lung cancer, as an outcome, and smoking as the only explanatory variable. It was also assumed that there are other factors that were likely to cause lung cancer but these were either unobserved, difficult to measure, or were totally unknown. These may include some genetic factors, personal differences in diet, childhood exposures, lifestyle, or other individual specific factors. In a statistical model, the relationship may be expressed as:
The maximum likelihood (ML) estimate equation for such a model may be written as:
2.2. Estimation of Bias, Using Taylor Series Expansion: The Uncorrelated Case
We consider the situation where sampled observations were identically independently distributed (IID), expected value of y, E(Y) exists, X was a binary variable (0/1), and ϵ was a random variable that was unobserved but has an influence on the response y. The unobserved term was assumed to follow a normal distribution, with mean zero and a variance . The maximum likelihood estimator,
for the logistic model, with one explanatory variable x, was first obtained ignoring unobserved heterogeneity; the effect of unobserved heterogeneity, ϵ on the response was then brought in, and the expected value of the (ML) estimator was evaluated again. Taylor series expansion was used and the estimates were compared (ignoring unobserved heterogeneity and including the influence of the term on the response y). The derivation showed that the (ML) estimator is biased due to un-modeled unobserved heterogeneity and the bias may be approximated by a first order term δ1, of Taylor series as follows:
2.3. Simulation
Simulation was used to examine situations that were thought to affect the severity of bias. Data was simulated from a logistic model with a response y, an explanatory variable x, and an extra term representing unobserved heterogeneity. A Fortran program was written to generate data from a logit function of the form:
4. Results
3.1. Theoretical Bias: The Effect of Position of the Two Probabilities
We examined the size of bias with focus on the importance of first-, second-, and third-order terms in the overall bias approximation. Table illustrates the contribution of each term under varying situations. Different positions of the two probabilities were examined, while the variance of unobserved heterogeneity was kept fixed at 1.0. The positions considered include: (i) the two probabilities lie on one side of 0.5 of the logistic curve; (ii) one of the probabilities is 0.5; and (iii) the two probabilities lie on the opposite side of 0.5.
Table 1 Theoretical contribution of first-, second-, and third-order terms to the approximation of bias due to unobserved heterogeneity, with variance
For all three situations, the first term δ1 was dominant, the second-order term is of less importance, and the third-order term has a very small contribution that may be safely ignored at almost all situations. There are, however, special situations where the second-order term becomes more important, for example, as one probability approaches 0.9.
3.2. Simulation Results: Theoretical and Numerical Bias
Table , reports the average estimates of 100 simulations, each based on a sample of size 1,000. The table examined various positions for the two probabilities, and a range of variance for unobserved heterogeneity (1.0–0.01) including the special case where unobserved heterogeneity, was almost constant ( = 0.01). Three situations regarding the position of the two probabilities were covered as described earlier. A comparison was then drawn between the theoretical bias, and the numerical bias, which was calculated from the simulation. The 95% confidence intervals for the simulated bias were reported and the number of times the confidence intervals of the estimate
, fail to cover the true parameter β was reported as percentage rejected (Rej%). The main findings may summarized as:
1. | The bias was well approximated by the first-order term at most of the situations considered. | ||||
2. | The bias becomes more serious as the difference between the two probabilities gets larger, especially as one probability gets closer to 0.9 (the situation with one probability approaching 0.1 is identical) or where the two probabilities lie further apart at the two sides of the logistic curve. | ||||
3. | The bias was more serious for relatively large variance of unobserved heterogeneity, 1.0 and 0.75, from the range of values considered. | ||||
4. | Contributions from the second-order term to the bias approximation become of some importance at special situation where the difference between the two probabilities was large, and the variance of unobserved heterogeneity was relatively large. | ||||
5. | For small variance of unobserved heterogeneity, the bias was small and the percentage of rejections was modest. | ||||
6. | For the special case of unobserved heterogeneity with variance =0.01, the bias either disappeared or became negligible, and the percentage of estimates outside the coverage property of |
Table 2 A comparison between theoretical bias and numerical (simulation) bias, for the (MLH) estimator
, of the logistic model, for various probabilities, and for a range of variance (0.01–1.0) for the unobserved heterogeneity term ϵ
4. Discussion
Much attention in the 1980's, and after, was given to the asymptotic bias of the maximum likelihood (ML) estimators of binary response regression models, that are widely used to describe associations between binary outcome and explanatory variables in trials and surveys. In general, ML estimators may not hold in small and finite samples, as shown by Anderson and Richardson (Citation1979) where a simulation was used to investigate bias of the logistic model estimates, the study found that bias can be substantial if the sample size is small, a formulae for correction was developed. Another similar study (Griffiths et al., Citation1987) examined the bias and other sample properties such as mean square error based on three alternative covariance matrix estimators for the Probit model, also reached the same conclusion with regard to the bias. A simpler formula using Taylor series expansion for correction of bias in logistic ML estimate was also developed by Copas (Citation1988). For exponential family such as the logistic model, the bias was of order O(1/n) suggesting that for large samples it was negligible relative to the standard errors of the estimates, the bias was treated by Jeffery's prior' as reported in McCullagh and Nelder (Citation1989) and Firth (Citation1992). A set of GLIM macros was developed to reduce bias (Firth, Citation1993; Steyerberg and Eijkemans Citation2004), but the reduction achieved was reported to be small.
Another issue of importance to the (ML) estimation was the deviation of data from the assumed identical independent distribution, that was addressed in survey methodology and procedures for correction of estimates were developed (Skinner and Smith, Citation1989).
Of no less importance was the problem of unobserved heterogeneity, although awareness of the problem has recently increased (Aprahamian et al., Citation2007; Arana and Leon Citation2006; Cramer Citation2007), but still many influential articles continue to report important findings ignoring the possibility of any impact of unobserved heterogeneity on these findings. In practice, in almost any biological investigation, there are factors (exogenous or endogenous, independent of a biological process, or part of it, time varying or time invariant, personal or contextual) that would be unobserved. Using nonlinear models under such situations lead to biased estimates of population parameters.
Here we present the case with unobserved heterogeneity that was linked to individuals (for example, taste, charisma, emotions), another scenario is where unobserved heterogeneity was correlated, that may occur with repeated measurements within individual, or due to clusters such as household (family related gene, for example), more details on the correlated case were reported in an earlier study (Ayis, Citation1995) where it was shown that the leading term of bias approximation for the correlated case was the same as that for the non correlated. An extra term, due to replications, however, becomes important if the number of clusters was small, and where the two probabilities lie further apart within the logistic curve.
While there are situations where estimates from the logistic model may be fairly robust to unobserved heterogeneity, there are others where the problem deserves more attention. For situations with outcomes such as fertility or incidence of disease, where all of the probabilities were on the same side of 0.5, the potential for bias was there, but perhaps not as bad as where extreme probabilities occur in both tails, for example (p 0 ≤ 0.2, p 1 ≥ 0.8). The work by Copas (Citation1988) is also relevant to the latter situation of extreme probabilities, although the assumption was that extreme values occur due to mis-recording, that is where the values of the response “y” was being transposed in error between 0 and 1, rather than due to the nature of the association between the response and the explanatory variable we present. Mont Carlo simulation was used to examine the sensitivity of different binary response models to such extreme values of probabilities, a model was proposed to allow for robust estimation where a small number of outcomes was being mis-recorded, and techniques for diagnostics where developed. For situations like the one we present in this study, extreme values will be more common, but detection of such values may help in assessing the quality of the estimates and possibly in whether to use alternative methods at situations were the bias is serious.
The misspecification bias of the logistic model, due to unobserved heterogeneity described in this study, is similar to the misspecification bias due to incorrectly assuming the error term was logistically distributed when it was not, as described in Horowitz (Citation1993), where the effect on estimates was measured using simulation. The bias was found to be small as long as the assumed error distribution has the same qualitative shape as the true distribution (unimodal for the logistic case considered) and more serious when the true distribution of the error term was bimodal or heteroskedastic. These findings suggest the need to explore the effect of other forms of distribution on the bias derived in this study. Similar findings were shown by Arana and Leon (Citation2005) where a Monte-Carlo simulation was used to test the performance of a Bayesian mixture normal distribution (semi parametric model), with other parametric models (including the logit) and nonparametric models, using alternative assumptions for the error distribution and using different sample sizes, when data exhibit unobserved heterogeneity. The mean square error (MSE) in all models and for all sample sizes was found to be considerably large reflecting the difficulty in modeling this type of data. The Bayesian specification, however, performed better than the competing models for small as well as for large samples, and substantially reduces the bias and the MSE, the improvements in bias was larger for small samples.
Misspecification bias due to unobserved heterogeneity can seriously affect estimates from the logistic model as well as other binary regression models. Using panel data and suitable random effect models that allow for individual fixed effect adjustment is a solution, but obtaining such data may not always be easy. Attempting other semiparametric models such as Bayesian normal mixture model seems to be an appealing approach, especially as suitable software are rapidly developing. Further work is needed to examine the impact on estimates in real situations where there are several explanatory variables often correlated. Methods of detection may also be developed to assess the need for alternative more flexible models, at situations where unobserved heterogeneity is likely to have more serious impact on the estimates due to data structure, before drawing inference that might be misleading.
Appendix A
Consider a MLH estimator, for a logistic model with binary response y and a single binary explanatory variable x,
Corollary A.1
Consider the expectations of corresponding terms of log n
1 (1 − ) of Eq. (A.1). These may be written as:
We apply the same procedures for terms associated with and 1 −
, from Eq. (A.1) and define similar functions of p
0 at this stage.
Now consider Term II,
Proof
We first restrict the bias to include only the first-order terms. The proof comes directly by substituting in Eq. (A.1), terms from I, from Eqs. (A.24), and complementary terms from (A.27), plus other similar terms for, log(n
1
) and log(n
1 (1 −
)), the later terms are identical to the functions of
and (1 −
), but they are functions of
and (1 −
) and they take different signs as Eq. (A.1) showed. The expectation of
may then be written as:
If term III was also considered, components from Eqs. (A.39) and (A.40), and similar terms relevant to and (1 −
), were also brought and substituted in Eq. (A.1), by procedures similar to those used in manipulating and including I and II, the expected value E
1 {E
2 (
)} may be written as:
Acknowledgments
I am grateful to professor D. Holt for the initiatives, advice, and suggestions he gave to the theoretical and simulation investigations. Much thanks to Dr. Marie South and Dr. Peter Egger for considerable help with the FORTRAN programs used.
Notes
Note: 95% C.I: upper and lower 95% confidence intervals for the simulated bias. Rej%: the number of times, in percentage that the true parameter β, lied outside the 95% confidence intervals of the simulated estimator .
References
- Agresti , A. ( 2002 ). Categorical Data Analysis . Probability and Statistics . New York : Wiley-Interscience .
- Anderson , J. A. , Richardson , S. C. ( 1979 ). Logistic discrimination and bias correction in maximum likelihood estimation . Technometrics 21 : 71 – 78 .
- Aprahamian , F. , Chanel , O. , Luchini , S. ( 2007 ). Modeling starting point bias as unobserved heterogeneity in contingent valuation surveys: an application to air pollution . Amer. J. Agri. Econ. 89 : 533 – 547 .
- Arana , J. E. , Leon , C. J. ( 2005 ). Flexible mixture distribution modeling of dichotomous choice contingent valuation with heterogenity 250541 . J. Environm. Econ. Manage. 50 : 170 – 188 .
- Arana , J. E. , Leon , C. J. ( 2006 ). Modelling unobserved heterogeneity in contingent valuation of health risks . Appl. Econ. 38 : 2315 – 2325 .
- Ayis , S. A. M. ( 1995 ). Modelling Unobserved Heterogeneity: Theoretical and Practical Aspects . Southampton, UK : University of Southampton .
- Copas , J. B. ( 1988 ). Binary regression models for contaminated data . J. Roy. Statist. Soc. Ser. B (Methodological) 50 : 225 – 265 .
- Cramer , J. S. ( 2007 ). Robustness of logit analysis: unobserved heterogeneity and mis-specified disturbances . Oxford Bull. Econ. Statist. 69 : 545 – 555 .
- Firth , D. ( 1992 ). Bias reduction, the Jeffreys prior and GLIM . In: Fahrmeir , L. , Francis , B. , Gilchrist , R. , eds. Advances in GLIM and Statistical Modelling . New York : Springer , pp. 91 – 100 .
- Firth , D. A. V. I. ( 1993 ). Bias reduction of maximum likelihood estimates . Biometrika 80 : 27 – 38 .
- Gail , M. H. , Wieand , S. , Piantadosi , S. ( 1984 ). Biased estimates of treatment effect in randomized experiments with nonlinear regressions and omitted covariates . Biometrika 71 : 431 – 444 .
- Griffiths , W. E. , Hill , R. C. , Pope , P. J. ( 1987 ). Small sample properties of probit model estimators . J. Amer. Statist. Assoc. 82 : 929 – 937 .
- Horowitz , J. L. ( 1993 ). Semiparametric and nonparametric estimation of quantal response models . In: Maddala , G. S. , Rao , C. R. , Vinod , H. D. , eds. Handbook of Statistics . Amsterdam : Elsevier .
- Lee , S. , Lee , S. ( 2003 ). Testing heterogeneity for frailty distribution in shared frailty model . Commun. Statist. Theor. Meth. 32 : 2245 – 2253 .
- McCullagh , P. A. , Nelder , J. A. ( 1989 ). Generalized Linear Models , 2nd ed. . Monographs on Statistics and Applied Probability . London : Chapman Hall .
- Mosley , W. H. , Chen , L. C. ( 2003 ). An analytical framework for the study of child survival in developing countries. 1984 . Bull. World Health Organ. 81 : 140 – 145 .
- Peters , B. L. ( 2004 ). Is there a wage bonus from drinking? Unobserved heterogeneity examined . Appl. Econ. 36 : 2299 – 2315 .
- Skinner , C. J. , Holt , D. , Smith , T. M. F. (eds.). ( 1989 ). Analysis of Complex Surveys . Chichester : John Wiley Sons, Ltd .
- Steyerberg , E. W. , Eijkemans , M. J. C. ( 2004 ). Heterogeneity bias: the difference between adjusted and unadjusted effects . Med. Deci. Making 24 : 102 – 104 .
- World Health Organization ( 2001 ). International Classification of Functioning, Disability and Health: ICF (Geneva ed.).
- Zohoori , N. , Savitz , D. A. ( 1997 ). Econometric approaches to epidemiologic data: Relating endogeneity and unobserved heterogeneity to confounding . Ann. Epidemio. 7 : 251 – 257 .