Toward Optimal Variance Reduction in Online Controlled Experiments

Abstract

We study optimal variance reduction solutions for count and ratio metrics in online controlled experiments. Our methods apply flexible machine learning tools to incorporate covariates that are independent from the treatment but have predictive power for the outcomes, and employ the cross-fitting technique to remove the bias in complex machine learning models. We establish CLT-type asymptotic inference based on our estimators under mild convergence conditions. Our procedures are optimal (efficient) for the corresponding targets as long as the machine learning estimators are consistent, without any requirement for their convergence rates. In complement to the general optimal procedure, we also derive a linear adjustment method for ratio metrics as a special case that is computationally efficient and can flexibly incorporate any pretreatment covariates. We evaluate the proposed variance reduction procedures with comprehensive simulation studies and provide practical suggestions regarding commonly adopted assumptions in computing ratio metrics. When tested on real online experiment data from LinkedIn, the proposed optimal procedure for ratio metrics can reduce up to 80% of variance compared to the standard difference-in-mean estimator and also further reduce up to 30% of variance compared to the CUPED approach by going beyond linearity and incorporating a large number of extra covariates.

Keywords:

• A/B test
• Causal inference
• Clustered randomized experiments
• Covariate adjustment
• Ratio metrics
• Semiparametric efficiency

Supplementary Materials

Deferred results and technical proofs

Deferred theoretical results (inference and optimality) for Algorithm 3, as well as proofs of all theoretical results in the article. (pdf)

Reproduction codes

Reproduction codes for the simulation studies in the article. (python codes and readme file)

Acknowledgments

This work is done during the first author’s internship at LinkedIn Applied Research team. The authors would like to thank Dominik Rothenhäusler for helpful discussions and thank Weitao Duan, Rina Friedberg, Reza Hosseini, Juanyan Li, Min Liu, Jackie Zhao, Sishi Tang and Parvez Ahammad for their suggestions and feedbacks. The authors would also like to thank the Editor, AE and the anonymous referee for their valuable comments.