Factorial Designs for Online Experiments

Abstract

Online experiments and specifically A/B testing are commonly used to identify whether a proposed change to a web page is in fact an effective one. This study focuses on basic settings in which a binary outcome is obtained from each user who visits the website and the probability of a response may be affected by numerous factors. We use Bayesian probit regression to model the factor effects and combine elements from traditional two-level factorial experiments and multiarmed bandits to construct sequential designs that embed attractive features of estimation and exploitation.

KEYWORDS:

• A/B testing
• Bayesian analysis
• Online experiments
• Probability matching
• Sequential design
• Thompson sampling

Acknowledgments

We wish to thank the referees and the associate editor. Their constructive feedback led to many improvements in this article.