A Confidence Interval for the Difference Between Standardized Regression Coefficients

Abstract

Researchers are often interested in comparing predictors, a practice commonly done via informal comparisons of standardized regression slopes. However, formal interval-based approaches offer advantages over informal comparison. Specifically, this article examines a delta-method-based confidence interval for the difference between two standardized regression coefficients, building upon previous work on confidence intervals for single coefficients. Using Monte Carlo simulation studies, the proposed approach is evaluated at finite sample sizes with respect to coverage rate, interval width, Type I error rate, and statistical power under a variety of conditions, and is shown to outperform an alternative approach that uses the standard covariance matrix found in regression textbooks. Additional simulations evaluate current software implementations, small sample performance, and multiple comparison procedures for simultaneously testing multiple differences of interest. Guidance on sample size planning for narrow confidence intervals, an R function to conduct the proposed method, and two empirical demonstrations are provided. The goal is to offer researchers a different tool in their toolbox for when comparisons among standardized coefficients are desired, as a supplement to, rather than a replacement for, other potentially useful analyses.

Keywords:

• Standardized regression coefficient
• confidence interval
• multiple regression

Notes

1 Relatedly, even critics note that standardized coefficients can be useful in the social sciences. For example, King (1986) wrote that standardized variables may be “the more natural measure(s) for some concepts, particularly for some psychological scales” (p. 673).

2 The formulas underlying the delta method covariance matrix differ for fixed and random predictors (Yuan & Chan, 2011), and the formulation used here assumes the latter.

3 ${\mathbf{\Sigma }}_{\mathit{\text{XX}}}$ is the population covariance matrix of predictors and ${\mathbf{\sigma }}_{\mathit{\text{Xy}}}$ is the population vector of covariances between the predictors and Y.

4 For model $R^2=.80,$ results were poorest when the individual $\beta$ values were larger in absolute value, which occurred when the predicted outcome scores were strongly associated with the last principal component of the predictors. Jones & Waller (2013) generated the population data in a unique way, such that various $\mathbf{\beta }$ vectors satisfying the same $R^2$ were generated with different geometric orientations relative to the eigenvectors of the predictor correlation matrix.

5 Recalling Equation (5), Rencher & Schaalje (2008) note, “in effect, ${\mathbf{R}}_{\mathit{\text{XX}}}$ and ${\mathbf{r}}_{\mathit{\text{Xy}}}$ are the covariance matrix and covariance vector for standardized variables (p. 251).

6 More technically, the covariance matrix is $\mathbf{J}\mathbf{\Gamma }{\mathbf{J}}^{⊺}/N+O(1/N),$ wherein the approximation improves as N grows.

7 Yuan and Chan’s derivations assumed at least an elliptical distribution for the data-generating model, of which the normal distribution is a special case.

8 This link also includes the simulated data for the sample size analysis (described in the Sample Size Planning section, which follows Results) as well as the function code for the Diff.Beta R function (described in the Software and Empirical Analysis sections that follow), and the Supplemental Materials document.

9 Negatively-signed versions of the resulting $\mathbf{\beta }$ vectors yield identical covariance matrices $\mathbf{\Sigma }$ with the exception of negatively-signed ${\mathbf{\rho }}_{\mathit{\text{Xy}}}$ and performance is the same. Thus, only positive versions of these $\mathbf{\beta }$ vectors were included.

10 For the sake of brevity, only data generation conditions with positively correlated predictors are shown in the main text, given that the problematic conditions for the standard method were the same and that delta method continued to perform well in these conditions.

11 The R code for these implementations, including specifying the phantom variable in lavaan model syntax, is available at the OSF link provided at the beginning of the method section. More details on phantom variables and related approaches in lavaan can be found in Cheung (2009), Kwan & Chan (2011), and Klopp (2020).

12 Methods (3) and (5) will be shown to perform equivalently, so only (3) was further examined.

13 Method (1) is a test only, so only Type I error rate was examined. Due to computation time, simulation 2A used 5000 replications.

14 Because methods (3) and (5) were identical, only the former was included in the Simulation 2B.

15 Note that 0.6 is not shown for ${\rho }_{\mathit{\text{XX}}}=.5,$ given that the resulting covariance matrix is not positive definite.

16 Link: https://github.com/sfander312/DiffBeta.

17 The original code can be found in Appendix B of Jones & Waller (2013). Note that there appears to be a typo in the original code, wherein the matrix multiplication when computing $\mathbf{\Gamma }$ as in Equation (12) (Kp.1ft) should be a Kronecker product. An R package implementing Jones & Waller’s method for a single coefficient has very recently been developed (Pesigan et al., 2023), though the present R function was developed prior to the R package (and focuses on the difference in coefficients rather than a single coefficient).

18 When only one interval is of interest, MCP=“no” should be specified.

19 OSF link: <ul>https://osf.io/fzpmg/</ul>.

R packages can be installed from GitHub using the devtools package. To install and load DiffBeta, use the following code: install.packages(“devtools”); library(devtools); install_github(“sfander312/DiffBeta”); library(DiffBeta).

20 The dataset is available through the Inter-University Consortium for Political and Social Research (ICPSR), at https://doi.org/10.3886/E119828V1

21 The dataset (Rateprof) is available via the alr4 package in R (Fox & Weisberg, 2019).

22 Relatedly, see Jones & Waller (2015) for an asymptotically distribution free method for individual standardized regression coefficients that does not make the multivariate normality assumption.

Additional information

Funding

The author(s) reported there is no funding associated with the work featured in this article.