Influence Functions of Iteratively Reweighted Least Squares Estimators

Abstract

The iteratively reweighted least squares algorithm is routinely employed to evaluate robust regression estimates. The importance of beginning the algorithm with a robust estimator of the unknown parameters is often stressed. The precise influence of the initial estimator on subsequent iterates or the limit of such estimators, however, does not seem to have been calculated previously. Because robust regression involves the downweighting of specific points having large rescaled residuals and/or large leverages in the design space, it is natural to think in terms of the weights themselves, rather than some less intuitive function such as the η function of generalized M estimators. Therefore, we consider multiple linear regression by the method of weighted least squares, where the weights are estimated quantities depending on both position in the design space and the residual relative to an initial regression estimator. The influence function of the weighted least squares estimator is shown to depend both on the weights and on the influence function of the initial estimator. When the weights are iteratively reestimated, convergence of the corresponding estimates and their influence functions depends on the magnitude of the iteration derivative, which is the derivative of the k + 1st iterate with respect to the kth iterate. Conditions are given on the weights for such convergence and for boundedness of the limiting influence function. For the class of generalized M estimators, the influence function of the k + 1st iterate is shown to be a convex matrix mixture of the influence functions of the kth iterate and the fully iterated M estimate. A brief numerical study for Huber M estimators suggests that the iteration derivative determines not only the speed of convergence of the sequence of influence functions corresponding to iterated reweighted least squares but the speed of convergence of the iterated estimates themselves.

Key Words:

• Bounded influence
• Generalized M estimator
• One-step estimator
• Robust regression
• Statistical functional
• Weighted least squares