Locally Optimal Design for A/B Tests in the Presence of Covariates and Network Dependence

Abstract

A/B test, a simple type of controlled experiment, refers to the statistical procedure of experimenting to compare two treatments applied to test subjects. For example, many IT companies frequently conduct A/B tests on their users who are connected and form social networks. Often, the users’ responses could be related to the network connection. In this article, we assume that the users, or the test subjects of the experiments, are connected on an undirected network, and the responses of two connected users are correlated. We include the treatment assignment, covariate features, and network connection in a conditional autoregressive model. Based on this model, we propose a design criterion that measures the variance of the estimated treatment effect and allocate the treatment settings to the test subjects by minimizing the criterion. Since the design criterion depends on an unknown network correlation parameter, we adopt the locally optimal design method and develop a hybrid optimization approach to obtain the optimal design. Through synthetic and real social network examples, we demonstrate the value of including network dependence in designing A/B experiments and validate that the proposed locally optimal design is robust to the choices of parameters. Supplementary materials for this article are available online.

KEYWORDS:

• A/B test
• Conditional autoregressive model
• Controlled experiments
• Covariates
• Optimal design

Supplementary Materials

The supplementary materials include proofs, derivations, and extra examples. They also include the codes for all the examples.

Division of Mathematical Sciences;

Funding

This research is supported by a U.S. National Science Foundation grant DMS-1916467.

Acknowledgments

The authors thank the Editor, the Associate Editor, and two reviewers for their valuable comments through the reviewing process.

Additional information

Funding

This research is supported by a U.S. National Science Foundation grant DMS-1916467.