Post-hoc selection of covariates in randomized experiments

Abstract

Monte Carlo methods are used to compere a number of adaptive strategies for deciding which of several covariates to incorporate into the analysis of a randomized experiment.Sixteen selection strategies in three categories are considered: 1)select covariates correlated with the response, 2)select covariates with means differing across groups, and 3)select covariates with means differing across groups that are also correlated with the response. The criteria examined are the type I error rate of the test for equality of adjusted group means and the variance of the estimated treatment effect. These strategies can result in either inflated or deflated type I errors, depending on the method and the population parameters. The adaptive methods in the first category some times yieldpoint estimates of the treatment effect more precise than estimators derive dusing either all or none of the covariates.

Keywords:

• analysis of covariance
• variable selection
• Monte Carlo studies
• stepwise regression
• Preliminary tests