Toward Optimal Variance Reduction in Online Controlled Experiments

References

• Andrews, D. W. (1994), “Empirical Process Methods in Econometrics,” Handbook of Econometrics, 4, 2247–2294.
   Google Scholar
• Athey, S., and Imbens, G. W. (2016), “Recursive Partitioning for Heterogeneous Causal Effects,” Proceedings of the National Academy of Sciences, 113, 7353–7360. DOI: 10.1073/pnas.1510489113.
   PubMed Web of Science ®Google Scholar
• Athey, S., and Wager, S. (2019), “Estimating Treatment Effects with Causal Forests: An Application,” Observational Studies, 5, 36–51. DOI: 10.1353/obs.2019.0001.
   Google Scholar
• Bickel, P. J., Klaassen, C. A., Ritov, Y., and Wellner, J. A. (1993), Efficient and Adaptive Estimation for Semiparametric Models, Baltimore: Johns Hopkins University Press.
   Google Scholar
• Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. (2018), “Double/Debiased Machine Learning for Treatment and Structural Parameters,” The Econometrics Journal, 21, 1–68. DOI: 10.1111/ectj.12097.
   Web of Science ®Google Scholar
• Chernozhukov, V., Demirer, M., Duflo, E., and Fernández-Val, I. (2017), “Generic Machine Learning Inference on Heterogenous Treatment Effects in Randomized Experiments,” arXiv preprint arXiv:1712.04802.
   Google Scholar
• Cohen, P. L., and Fogarty, C. B. (2020), “No-harm Calibration for Generalized Oaxaca-Blinder Estimators,” arXiv preprint arXiv:2012.09246.
   Google Scholar
• Deng, A., Lu, J., and Litz, J. (2017), “Trustworthy Analysis of Online a/b Tests: Pitfalls, Challenges and Solutions,” in Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, pp. 641–649.
   Google Scholar
• Deng, A., Xu, Y., Kohavi, R., and Walker, T. (2013), “Improving the Sensitivity of Online Controlled Experiments by using Pre-experiment Data,” in Proceedings of the sixth ACM international conference on Web search and data mining, pp. 123–132. DOI: 10.1145/2433396.2433413.
   Google Scholar
• Freedman, D. A. (2008), “On Regression Adjustments to Experimental Data,” Advances in Applied Mathematics, 40, 180–193. DOI: 10.1016/j.aam.2006.12.003.
   Web of Science ®Google Scholar
• Friedman, J. H. (1991), “Multivariate Adaptive Regression Splines,” The Annals of Statistics, 19, 1–67. DOI: 10.1214/aos/1176347963.
   Web of Science ®Google Scholar
• Green, D. P., and Vavreck, L. (2008), “Analysis of Cluster-Randomized Experiments: A Comparison of Alternative Estimation Approaches,” Political Analysis, 16, 138–152. DOI: 10.1093/pan/mpm025.
   Web of Science ®Google Scholar
• Guo, K., and Basse, G. (2021), “The Generalized Oaxaca-Blinder Estimator,” Journal of the American Statistical Association, 1–35. DOI: 10.1080/01621459.2021.1941053.
   PubMed Web of Science ®Google Scholar
• Guo, Y., Coey, D., Konutgan, M., Li, W., Schoener, C., and Goldman, M. (2021), “Machine Learning for Variance Reduction in Online Experiments,” arXiv preprint arXiv:2106.07263.
   Google Scholar
• Hahn, J. (1998), “On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects,” Econometrica, 66, 315–331. DOI: 10.2307/2998560.
   Web of Science ®Google Scholar
• Hosseini, R., and Najmi, A. (2019), “Unbiased Variance Reduction in Randomized Experiments,” arXiv preprint arXiv:1904.03817.
   Google Scholar
• Imai, K., and Ratkovic, M. (2013), “Estimating Treatment Effect Heterogeneity in Randomized Program Evaluation,” The Annals of Applied Statistics, 7, 443–470. DOI: 10.1214/12-AOAS593.
   Web of Science ®Google Scholar
• Imbens, G. W., and Rubin, D. B. (2015), Causal Inference in Statistics, Social, and Biomedical Sciences, Cambridge: Cambridge University Press.
   Google Scholar
• Kennedy, E. H. (2020), “Optimal Doubly Robust Estimation of Heterogeneous Causal Effects,” arXiv preprint arXiv:2004.14497.
   Google Scholar
• Kohavi, R., Tang, D., and Xu, Y. (2020), Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge: Cambridge University Press.
   Google Scholar
• Künzel, S. R., Sekhon, J. S., Bickel, P. J., and Yu, B. (2019), “Metalearners for Estimating Heterogeneous Treatment Effects using Machine Learning,” Proceedings of the National Academy of Sciences, 116, 4156–4165. DOI: 10.1073/pnas.1804597116.
   PubMed Web of Science ®Google Scholar
• Lin, W. (2013), “Agnostic Notes on Regression Adjustments to Experimental Data: Reexamining Freedman’s Critique,” The Annals of Applied Statistics, 7, 295–318. DOI: 10.1214/12-AOAS583.
   Web of Science ®Google Scholar
• Middleton, J. A., and Aronow, P. M. (2015), “Unbiased Estimation of the Average Treatment Effect in Cluster-Randomized Experiments,” Statistics, Politics and Policy, 6, 39–75. DOI: 10.1515/spp-2013-0002.
   Google Scholar
• Nichols, A. (2007), “Causal Inference with Observational Data,” The Stata Journal, 7, 507–541. DOI: 10.1177/1536867X0800700403.
   Web of Science ®Google Scholar
• Nie, X., and Wager, S. (2020), “Quasi-Oracle Estimation of Heterogeneous Treatment Effects,” Biometrika, forthcoming. DOI: 10.1093/biomet/asaa076.
   Google Scholar
• Robins, J. M., Rotnitzky, A., and Zhao, L. P. (1994), “Estimation of Regression Coefficients When Some Regressors are not Always Observed,” Journal of the American Statistical Association, 89, 846–866. DOI: 10.1080/01621459.1994.10476818.
   Web of Science ®Google Scholar
• Schuler, M. S., and Rose, S. (2017), “Targeted Maximum Likelihood Estimation for Causal Inference in Observational Studies,” American Journal of Epidemiology, 185, 65–73. DOI: 10.1093/aje/kww165.
   PubMed Web of Science ®Google Scholar
• Van der Vaart, A. W. (2000), Asymptotic Statistics (Vol. 3), Cambridge: Cambridge University Press.
   Google Scholar
• Van Der Vaart, A. W., van der Vaart, A., van der Vaart, A. W., and Wellner, J. (1996), Weak Convergence and Empirical Processes: With Applications to Statistics, New York: Springer.
   Google Scholar
• Wager, S., and Athey, S. (2018), “Estimation and Inference of Heterogeneous Treatment Effects using Random Forests,” Journal of the American Statistical Association, 113, 1228–1242. DOI: 10.1080/01621459.2017.1319839.
   Web of Science ®Google Scholar
• Wu, C. F. J., and Hamada, M. S. (2009), Experiments: Planning, Analysis, and Optimization, Hoboken, NJ: Wiley.
   Google Scholar
• Yang, L., and Tsiatis, A. A. (2001), “Efficiency Study of Estimators for a Treatment Effect in a Pretest–Posttest Trial,” The American Statistician, 55, 314–321. DOI: 10.1198/000313001753272466.
   Web of Science ®Google Scholar