![MathJax Logo](/templates/jsp/_style2/_tandf/pb2/images/math-jax.gif)
Abstract
We study response surface designs using the generalized variance inflation factors for subsets as an extension of the variance inflation factors.
Public Interest Statement
Response surface designs are a mainstay in applied statistics. The variance inflation factors VIF are a measure of collinearity for a single variable in a linear regression model. The generalization to subsets of variables is the generalized variance inflation factor GVIF. This research introduces as a penalty measure for extending a linear response model to a response surface with the included quadratic terms. The methodology is demonstrated with case studies, and, in particular, it is shown that using GVIF, the H310 design can be improved for the standard global optimality criteria of
,
, and
.
1. Introduction
We consider a linear regression with
a full rank
matrix and
. The variance inflation factor
, Belsley (Citation1986), measures the penalty for adding one non-orthogonal additional explanatory variable to a linear regression model, and they can be computed as a ratio of determinants. The extension of
to a measure of the penalty for adding a subset of variables to a model is the generalized variance inflation factor
of Fox and Monette (Citation1992), which will be used to study response surface designs, in particular, as the penalty for adding the quadratic terms to the model.
2. Variance inflation factors
For our linear model , let
be the diagonal matrix with entries on the diagonal
. When the design has been standardized
, the
s are the diagonal entries of the inverse of
. That is, the
s are the ratios of the actual variances for the explanatory variables to the “ideal” variances had the columns of
been orthogonal. Note that we follow Stewart (Citation1987) and do not necessarily center the explanatory variables.
For our linear model , view
with
the
column of
and
the matrix formed by the remaining columns. The variance inflation factor
measures the effect of adding column
to
. For notational convenience, we demonstrate
with the last column
. An ideal column would be orthogonal to the previous columns with the entries in the off-diagonal elements of the
row and
column of
all zeros. Denote by
the idealized moment matrix
The s are the diagonal entries of
. It remains to note that the inverse,
, can be computed using cofactors
. In particular,
(1)
(1)
the ratio of the determinant of the idealized moment matrix to the determinant of the moment matrix
. This definition extends naturally to subsets and is discussed in the next section.
For an alternate view of the how collinearities in the explanatory variables inflate the model variances of the regression coefficients when compared to a fictitious orthogonal reference design, consider the formula for the model variance
where is the square of the multiple correlation from the regression of the
column of
on the remaining columns as in Liao and Valliant (Citation2012). The first term
is the model variance for
had the
explanatory variable been orthogonal to the remaining variables. The second term
is a standard definition of the
VIF as in Thiel (Citation1971).
3. Generalized variance inflation factors
In this section, we introduce the GVIFs as an extension of the classical variance inflation factors from Equation 1. For the linear model
, view
partitioned with
of dimension
usually consisting of the lower order terms and
of dimension
usually consisting of the higher order terms. The idealized moment matrix for the
partitioning of
is
Following Equation 1, to measure the effect of adding to the design
, that is for
, we define the generalized variance inflation factor as
(2)
(2)
as in Equation 10 of Fox and Monette (Citation1992), who compared the sizes of the joint confidence regions for for partitioned designs and noted when
that
. Equation 2 is in the spirit of the efficiency comparisons in linear inferences introduced in Theorems 4 and 5 of Jensen and Ramirez (Citation1993). A similar measure of collinearity is mentioned in Note 2 in Wichers (Citation1975), Theorem 1 of Berk (Citation1977), and Garcia, Garcia, and Soto (Citation2011). For the simple linear regression model with
, Equation 2 gives
with
the correlation coefficient as required. Fox and Monette (Citation1992) suggested that
contains the variables which are of “simultaneous interest,” while
contains additional variables selected by the investigator. We will set
for the constant and main effects and set
the (optional) quadratic terms with values from
.
Willan and Watts (Citation1978) measured the effect of collinearity using the ratio of the volume of the actual joint confidence region for to the volume of the joint confidence region in the fictitious orthogonal reference design. Their ratio is in the spirit of
as
is inversely proportional to the square of the volume of the joint confidence region for
. They also introduced a measure of relative predictability and they note: “The existence of near linear relations in the independent variables of the actual data reduces the overall predictive efficiency by this factor.” For a simple case study, consider the simple linear regression model with
,
, and
. The
prediction interval for
is
. If the model also includes
, then the
prediction interval for
is
demonstrating the loss of predictive efficiency due to the collinearity introduced by
.
For the partition of
with
of dimension
and
of dimension
, set
and denote the canonical moment matrix as(3)
(3)
with determinant
equivalently,
where .
In the case ,
,
is a
vector and the partitioned design
has
. From standard facts for the inverse of a partitioned matrix, for example, Myers (Citation1990, p. 459),
can be computed directly as
Table 1. CCD with parameter , canonical index
, and
We study the eigenvalue structure of in Appendix 1. Let
be the non-negative singular values of
. It is shown in Appendix 1 that an alternative formulation for
is
(4)
(4)
4. Quadratic model with ![](//:0)
![](//:0)
For the partitioning , the canonical moment matrix, Equation 3, has the identity matrices
,
down the diagonal and off-diagonal array
. For the quadratic model
and partitioning
, we have
From Equation 4, where
is the unique positive singular value of
. Denote
as the canonical index with . Surprisingly, many higher order designs also have the off-diagonal entry of the canonical moment matrix with a unique positive singular value with
with the collinearity between the lower order terms and the upper order terms as a function of the canonical index
.
5. Central composite and factorial designs for quadratic models ![](//:0)
![](//:0)
In this section, we compare the central composite design (CCD) of Box and Wilson (Citation1951) and the factorial design
The design points are shown in Table of Appendix 2. Both designs are
and use the quadratic response model
The CCD traditionally uses the value in four entries, while the factorial design uses the value
. To study the difference in the designs with these different values, we computed the GVIF to compare the “orthogonality” between the lower order terms
of dimension
and the higher order quadratic terms
of dimension
. The off-diagonal
entry of
from Equation 3 in Section 3 has the form
with , canonical index
and
as in the quadratic model case with
shown in the Section 4. For Table , if
, then
,
, and
. Surprisingly, the classical choice of
gives the largest value for
, that is the worst value, indicating the greatest collinearity between the lower and higher order terms, as noted in O’Driscoll and Ramirez (Citationin press).
6. Larger designs ![](//:0)
![](//:0)
We consider the quadratic response surface designs for(5)
(5)
with responses and with
partitioned into
with
the four lower order terms
and
the six quadratic terms
. Four popular designs are given in Appendix 2. They are the hybrid designs (
and
) of Roquemore (Citation1976) Tables and , the Box and Behnken (Citation1960)
design Table , and the CCD of Box and Wilson (Citation1951) Table .
For each design, we compute the canonical moment matrix. It is striking that, for all these designs, the off-diagonal
array in
has only one non-zero singular value with its square the canonical index
. It follows that
.
Table 2. Hybrid designs ,
, Box and Behnken
CCD
Table 3. Singular values for off-diagonal array of for BDD and SCD with
Table reports that the design is the most conditioned with respect to the GVIF with the least amount of collinearity between the lower and higher order terms.
7. More complicated designs with ordered singular values
Let be the minimal design of Box and Draper (Citation1974) BDD with
from Table , and let
be the small composite design of Hartley (Citation1959)
with
from Table for the quadratic response surface model
and
as in Equation (Equation5
(5)
(5) ). Let
and
be the non-negative singular values of the off-diagonal array for
and
, respectively. As
in Table , it follows that
showing less collinearity between the lower and higher order terms for the BDD design.
8. An improved H310 design
When the diagonal matrix in Equation 6 in Appendix 1 has only one non-zero entry, we have denoted the square of this value the canonical index. We extend this definition to the case when
has multiple positive singular values. The Frobenious norm for a rectangular matrix
is defined by
. For a design matrix
, we extend the definition of the canonical index with
. Alternatively,
as in Equation 7.
We examine, in detail, the design matrix
, Table in Appendix 2, with our attention to the value of
in row 2 for
. In succession, we will replace the values
by a free parameter and use
to determine an optimal value. For example, replacing the four entries which are
with
, we calculate the minimum value for
with
denoted
in Table . These values are within the four digit accuracy of the data. We performed a similar calculation with
using the four entries which are
; with
with the four entries which are
; with
with the eight entries which are
; and with
with the single entry
. The original design has
. The entries in the
design are given to four significant digits. With this precision, the original design is nearly optimal with respect to the canonical index
for the first five entries in Table . The sixth entry of
was not optimal with
with
, a magnitude value smaller.
Table 4. Optimal values for
Denote the “improved” design as the
design with the value of
. The “improved”
also has a unique positive singular value for the off-diagonal of
with its square the canonical index
. All of the standard design criteria favor the “improved”
design over the
design, which was originally constructed based on the rotatability criterion to maintain equal variances for predicted responses for points that have the same distance from the design center. As usual
,
, and
eigenvalues of
. The small relative changes
in the design criteria are shown in Table in Column 4.
Table 5. Design criteria for the “improved” with
the relative change
The abnormality of the second row in has been noted in Jensen (Citation1998) who showed that the design is least sensitive to the second row of
the row containing the value
.
9. Conclusions
The VIF measure the penalty for adding a non-orthogonal variable to a linear regression. The can be computed as a ratio of determinant as in Equation 1. A similar ratio criterion was studied by Fox and Monette (Citation1992) to measure the effect of adding a subset of new variables to a design and they dubbed it the generalized variance inflation factor
, Equation 2. We have noted the relationship between
and the singular values of the off-diagonal array in the canonical moment matrix and have used
to study standard quadratic response designs. The
design of Roquemorer (Citation1976) was shown not to be optimal with respect to
and an “improved”
design was introduced which was favored over
using the standard design criteria
,
, and
.
Additional information
Funding
Notes on contributors
Diarmuid O’Driscoll
Diarmuid O’Driscoll is the head of the Mathematics and Computer Studies Department at Mary Immaculate College, Limerick. He was awarded a Travelling Studentship for his MSc at University College Cork in 1977. He has taught at University College Cork, Cork Institute of Technology, University of Virginia, and Frostburg State University. His research interests are in mathematical education, errors in variables regression, and design criteria. In 2014, he was awarded a Teaching Heroes Award by the National Forum for the Enhancement of Teaching and Learning (Ireland).
Donald E. Ramirez
Donald E Ramirez is a full professor in the Department of Mathematics at the University of Virginia in Charlottesville, Virginia. He received his PhD in Mathematics from Tulane University in New Orleans, Louisiana. His research is in harmonic analysis and mathematical statistics. His current research interests are in statistical outliers and ridge regression.
References
- Belsley, D. A. (1986). Centering, the constant, first-differencing, and assessing conditioning. In E. Kuh & D. A. Belsley (Eds.),Model reliability (pp. 117–153). Cambridge: MIT Press.
- Berk, K. (1977). Tolerance and condition in regression computations. Journal of the American Statistical Association, 72, 863–866.
- Box, G. E. P., & Behnken, D. W. (1960). Some new three-level designs for the study of quantitative variables. Technometrics, 2, 455–475.
- Box, M. J., & Draper, N. R. (1974). On minimum-point second order design. Technometrics, 16, 613–616.
- Box, G. E. P., & Wilson, K. B. (1951). On the experimental attainment of optimum conditions. Journal of the Royal Statistical Society, Series B, 13, 1–45.
- Eaton, M. L. (1983). Multivariate statistics. New York, NY: Wiley.
- Fox, J., & Monette, G. (1992). Generalized collinearity diagnostics. Journal of the American Statistical Association, 87, 178–183.
- Garcia, C. B., Garcia, J., & Soto, J. (2011). The raise method: An alternative procedure to estimate the parameters in presence of collinearity. Quality and Quantity, 45, 403–423.
- Hartley, H. O. (1959). Smallest composite design for quadratic response surfaces. Biometrics, 15, 611–624.
- Jensen, D. R. (1998). Principal predictors and efficiency in small second-order designs. Biometrical Journal, 40, 183–203.
- Jensen, D. R., & Ramirez, D. E. (1993). Efficiency comparisons in linear inference. Journal of Statistical Planning and Inference, 37, 51–68.
- Liao, D., & Valliant, R. (2012). Variance inflation in the analysis of complex survey data. Survey Methodology, 38, 53–62.
- Myers, R. (1990). Classical and modern regression with applications (2nd ed.). Boston, MA: PWS-Kent.
- O’Driscoll, D., & Ramirez, D. E. (in press). Revisiting some design criteria ( under review).
- Roquemorer, K. G. (1976). Hybrid designs for quadratic response surfaces. Technometrics, 18, 419–423.
- Stewart, G. W. (1987). Collinearity and least squares regression. Statistical Science, 2, 68–84.
- Thiel, H. (1971). Principles of econometrics. New York, NY: Wiley.
- Wichers, R. (1975). The detection of multicollinearity: A comment. Review of Economics and Statistics, 57, 366–368.
- Willan, A. R., & Watts, D. G. (1978). Meaningful multicollinearity measures. Technometrics, 20, 407–412.
Appendix 1
We study the eigenvalue structure of . Let
be the non-negative singular values of
.
As with the canonical correlation coefficients Eaton (Citation1983), write the off-diagonal rectangular array of
as
with
and
orthogonal matrices and
the rectangular diagonal matrix with the non-negative singular values down the diagonal. Set
For notational convenience, we assume . The matrix
is orthogonal and transforms
into diagonal matrices:
(A1)
(A1)
with where
is the diagonal matrix of the non-negative singular values. Since
is orthogonal, this transformation has not changed the eigenvalues. To compute the determinant of
, convert the matrix in Equation 6 into an upper diagonal matrix by Gauss Elimination on
. This changes
of the
on the diagonal in rows
to
into
, and thus
with
The singular values of are the non-negative square roots of the eigenvalues of
denoted by
(A2)
(A2)
If the trace of the inverse of the matrix in Equation 6 is required, then we note that
with trace given by .
Appendix 2
Table A1. The lower order matrix for the CCD with center run with ,
and the lower order matrix for the factorial design with center run
Table A2. The lower order matrix for the hybrid design of Roquemore (Citation1976) with center run,
Table A3. The lower order matrix for the hybrid design of Roquemore (Citation1976) with center run,
Table A4. The lower order matrix for the Box and Behnken (Citation1960) design with center run,
Table A5. The lower order matrix for the Box and Wilson (Citation1951) CCD for with center run,
Table A6. The lower order matrix for the Box and Draper (Citation1974) minimal design (BDD) with center run,
Table A7. The Lower order matrix for the small composite design of Hartley (Citation1959) for
with center run,