Design choices in randomization tests that affect power

Abstract

We consider the problem of evaluating designs for a small-sample two-arm randomized experiment using the power of the one-sided randomization test as a criterion. Our evaluation assumes a linear response in one observed covariate, an unobserved component and an additive treatment effect where the model's randomness is due to different treatment allocations. It is well-known that the power depends on the allocations’ imbalance in the observed covariate. We show that power is additionally affected by two other design choices: the number of allocations and the degree of the allocations’ dependence. We prove that the more allocations, the higher the power and the lower the variability in power. Our theoretical findings and simulation studies show that the designs with the highest power provide thousands of highly independent allocations each providing nominal imbalance in the observed covariates. These high-powered designs exhibit less randomness than complete randomization and more randomness than recently proposed designs that employ numerical optimization. This advantage of high power is easily accessible to practicing experimenters via the popular rerandomization design and a greedy pair switching design, where both outperform complete randomization and numerical optimization. The tradeoff we find also provides a means to specify rerandomization’s imbalance threshold parameter.

Keywords:

• Experimental design
• statistical power
• restricted randomization
• rerandomization
• optimization

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This research was supported by Grant No 2018112 from the United States-Israel Binational Science Foundation (BSF).