Performance of a new ridge regression estimator

Abstract

Ridge regression estimator has been introduced as an alternative to the ordinary least squares estimator (OLS) in the presence of multicollinearity. Several studies concerning ridge regression have dealt with the choice of the ridge parameter. Many algorithms for the ridge parameter have been proposed in the statistical literature. In this article, a new method for estimating ridge parameter is proposed. A simulation study has been made to evaluate the performance of the proposed estimator based on the mean squared error (MSE) criterion. The evaluation has been done by comparing the MSEs of the proposed estimator with other well-known estimators. In the presence of multicollinearity, the simulation study indicates that under certain conditions the proposed estimator performs better than other estimators.

Keywords:

• Mean squared error
• Monte Carlo simulations
• Multicollinearity
• Ridge parameter
• Ridge regression

1 Introduction

Consider the standard model for multiple linear regression (Draper and Smith, 1998)(1) where is an n × 1 column vector of observations on the dependent variable, is an n × p fixed matrix of observations on the explanatory variables and is of full rank p (p ⩽ n), β is a p × 1 unknown column vector of regression coefficients, and is an n × 1 vector of random errors; , where denotes the n × n identity matrix and the prime denotes the transpose of a matrix. The variables are assumed to be standardized so that is in the form of correlation matrix, and the vector is the vector of correlation coefficients of the dependent variable with each explanatory variable. The ordinary least squares (OLS) estimator, , of the parameters is given by (Draper and Smith, 1998)(2) Clearly, is an unbiased estimator of β. Let denote the eigenvalues of . The mean squared error (MSE) of the components of is given by (Draper and Smith, 1998)(3) In application of multiple linear regression, the matrix might be nearly singular, that is, is small for some value of i. This is due to some inter-relation between the explanatory variables. The relation is technically called multicollinearity. The OLS estimator of regression coefficients tends to become “unstable” in the presence of multicollinearity. More precisely, the variance of the estimates of some of the regression coefficients becomes large. This is clear from (3)(3) .

Table 1 Estimated MSEs and the values of with p = 5 and n = 15.

γ	Estimators
	OLS	HK	HSL	NHSL
0.99	0.0369229	0.0328831	0.0336641	0.0268559	879.62
0.9	0.0270955	0.0248932	0.0250849	0.0213376	80.79
0.8	0.0204500	0.0191982	0.0192228	0.0173943	35.29
0.7	0.0159889	0.0153413	0.0153420	0.0132687	19.39

Table 2 Estimated MSEs and the values of with p = 10 and n = 25.

γ	Estimators
	OLS	HK	HSL	NHSL
0.99	0.0179593	0.0173544	0.0174193	0.0156275	2743.34
0.9	0.0123661	0.0121384	0.0121454	0.0106224	240.29
0.8	0.0088826	0.0087805	0.0087810	0.0083277	99.86
0.7	0.0070337	0.0069838	0.0069838	0.0066757	57.91

Table 3 Estimated MSEs and the values of with p = 20 and n = 30.

γ	Estimators
	OLS	HK	HSL	NHSL
0.99	0.01385120	0.01348810	0.01350310	0.00661062	23185.20
0.9	0.00877632	0.00867149	0.00867239	0.00414520	2002.15
0.8	0.00633292	0.00628823	0.00628828	0.00304895	830.08
0.7	0.00481253	0.00479185	0.00479185	0.00465087	477.50

Many attempts have been made to improve the OLS estimation procedure. In general, there are two approaches. One approach centers on finding (biased) estimators which have smaller MSE than the OLS estimators. Ridge regression, as well as many shrinkage type of estimators (Stein, 1960; Scolve, 1968"), is one example. This approach does not directly address itself to the issue of multicollinearity, even though multicollinearity is often the situation where the aforementioned procedures (or estimators) are used.

Among these estimators, the ridge estimator points indirectly to the issue of multicollinearity by constraining the length of the coefficient estimator (Hocking, 1976). In contrast, the second approach deals straightforward with the dependency nature of the explanatory variables. The principal components regression, as well as the latent root regression and the factor analysis approach, is one such example.

Hoerl and Kennard (1970a,b)" proposed the ridge estimator as an alternative to the OLS estimator for use in the presence of multicollinearity. The ridge estimator is given by(4) where denotes an identity matrix and k is a positive number known as ridge parameter. The corresponding MSE is given by(5) Though this estimator results in bias, for a certain value of k, it yields minimum MSE compared to the OLS estimator (Hoerl and Kennard, 1970a). However, the depends on unknown parameters k, β and , which can’t be calculated in practice, but k has to be estimated from the real data instead.

Several methods for estimating k have been proposed (Hoerl and Kennard, 1970a; Hoerl et al., 1975; McDonald and Galarneau, 1975; Lawless and Wang, 1976; Hocking et al., 1976; Wichern and Churchill, 1978; Nordberg, 1982; Saleh and Kibria, 1993; Singh and Tracy, 1999; Wencheko, 2000; Kibria, 2003; Khalaf and Shukur, 2005; Alkhamisi et al., 2006" and Alkhamisi and Shukur, 2007).

Alkhamisi and Shukur (2007) suggested a new approach to estimate the ridge parameter. They also proposed some new estimators by adding to some well-known estimators, where is the largest eigenvalue of . The authors used Monte Carlo experiments and the MSE criterion to compare the proposed estimators with some well-known estimators.

The purpose of this study is to apply the modification mentioned in Alkhamisi and Shukur (2007) to the estimator proposed by Hocking et al. (1976) in order to define a new estimator. A Monte Carlo comparison will be made using the MSE criterion to compare the performances of the proposed estimator with the OLS estimator and the estimators of Hocking et al., 1976 and Hoerl and Kennard, 1970 .

2 Methodology

It is convenient to express the regression model Eq. (1)(1) in the canonical form. Suppose that there exists an orthogonal matrix such that , where , and contains the eigenvalues of the matrix C, then the canonical form of the model (1)(1) is(6) where and . Then the OLS estimator is given as follows(7) and so we can write the ridge estimator as(8) where . Eq. (8)(8) is called the general form of ridge regression (Hoerl and Kennard, 1970a). It follows from Hoerl and Kennard (1970a) that the value of which minimizes the , where(9) is(10) where represents the error variance of model Eq. (1)(1) , is the ithelement of α.

Eq. (10(10) ) gives a value of that fully depends on the unknowns and and must be estimated from the observed data. Hoerl and Kennard (1970a) suggested the replacement of and by their corresponding unbiased estimators, that is,(11) where is the residual mean square estimate, which is an unbiased estimator of , and is the element of , which is an unbiased estimator of α. They found that the best method for achieving a better estimate is to use for all i, and they suggested k to be (or HK) where(12) If and α are known, then is sufficient to give ridge estimators having smaller MSE than the OLS estimator.

Hocking et al. (1976) suggested the following estimator, (or HSL), for k:(13) We now apply the modification mentioned in Alkhamisi and Shukur (2007) to the estimator proposed by Hocking et al. (1976), , to obtain our new estimator (or NHSL):(14) Since is grater than .

3 The simulation study

In this section, we use Monte Carlo simulation to investigate the properties of OLS, HK, HSL and NHSL. A comparison is then made based on the MSE criterion. Although many estimators can be considered in this simulation study (Kibria, 2003; Khalaf and Shukur, 2005"; Alkhamisi et al., 2006; Alkhamisi and Shukur, 2007" and Al-Hassan, 2008), we will only consider OLS, HK and HSL estimators and compare them with NHSL. We made these choices for the following reasons:

1.	Our interest herein lies in studying the properties of NHSL as an alternative of OLS in the presence of multicollinearity.

2.	HK estimator is the first ridge estimator that was proposed among all other estimators. Moreover, most of studies concerned with proposing new ridge estimators or comparing ridge estimators to each other take HK estimator in consideration.

3.	By construction, NHSL is a modified version of HSL, so we thought that it is necessary to make a comparison between them.

Therefore, we can say that it is convenient to make the comparison among OLS, HK, HSL and NHSL estimators. But, at the same time, we have to note that more investigation of NHSL is needed in future. This may be done by making comparisons between NHSL and other ridge estimators.

Following McDonald and Galarneau (1975), Wichern and Churchill (1978)", Gibbons (1981) and Kibria (2003), the explanatory variables were generated using the device(15) where are independent standard normal pseudo-random numbers, γ is specified so that the correlation between any two explanatory variables is given by , and p is the number of explanatory variables. The variables are then standardized so that and are in correlation forms. Different sets of correlation are considered corresponding to γ = 0.7, 0.8, 0.9 and 0.99. Using the condition number , it can be shown that these values of γ will include a wide range of low, moderate and high correlations between variables. The n observations for the dependent variable y are determined by(16) where are independent normal pseudo-numbers and is taken to be identically zero, and p is defined as in Eq. (15(15) ). We used three different sample sizes: 15, 25 and 30 with 5, 10 and 20 explanatory variables, respectively. These choices of p are taken to study the behavior of the estimators for small, moderate and large number of explanatory variables.

For each set of explanatory variables, one choice for the coefficient vectors is considered. The MSE function depends on the explanatory variables (through , on and on . It was noted (Newhouse and Oman, 1971) that if MSE is regarded as a function of with , k and the explanatory variables are fixed, then, subject to the constraint that , the MSE is minimized when is the normalized eigenvector corresponding to the largest eigenvalue of matrix . We didn’t use normalized eigenvectors corresponding to the smallest eigenvalue because the conclusion about the performance of estimators in both cases will not change greatly (Kibria, 2003).

For given values of p, n and γ, the experiment was repeated 1000 times by generating 1000 samples. For each replicate r (r = 1, 2,…, 1000), the values of k of different proposed estimators and the corresponding ridge estimators were calculated using(17) Then the MSEs for estimators are calculated as follows(18)

4 Results of the simulation study

In this section we present the results of our Monte Carlo experiments. Our primary interest herein lies in comparing the MSEs of the considered estimators. The main results of simulation are summarized in Tables 1–3 below. To compare the performances of the considered estimators, we calculate the MSEs of each one. We consider the estimator that leads to the minimum MSE to be the best. It is worth mentioning herein that we used the statistics package Minitab14 to do all calculations that were made in this article.

Looking at Tables 1–3, we can see that the HK, HSL and NHSL are better than the OLS, and the NHSL performs better than the HK and HSL. The results also reveal that for high correlations, i.e., when γ = 0.9 and 0.99, the HK and HSL perform almost equivalently. However, the HK produces somewhat lower MSEs than the HSL for all sets of correlation. Moreover, it is observed that for given n and p, the MSEs for all estimators increase as the correlation among the explanatory variables increases. In an opposite manner, for given, as the sample size and the number of explanatory variables increase, the MSEs of all estimators decrease.

5 Conclusion

In this article we have investigated the properties of a new proposed method for estimating the ridge parameter in the presence of multicollinearity. The investigation has been done using Monte Carlo experiments, where levels of correlation, the numbers of explanatory variables and the sample sizes have been varied. For each combination we have 1000 replications. The evaluation of our estimator has been done by comparing the MSEs of our proposed estimator with the OLS estimator and the estimators of Hocking et al. (1976) and Hoerl and Kennard (1970) . We found that our estimator uniformly dominates the other estimators.