Statistical Challenges in Online Controlled Experiments: A Review of A/B Testing Methodology

Abstract

The rise of internet-based services and products in the late 1990s brought about an unprecedented opportunity for online businesses to engage in large scale data-driven decision making. Over the past two decades, organizations such as Airbnb, Alibaba, Amazon, Baidu, Booking.com, Alphabet’s Google, LinkedIn, Lyft, Meta’s Facebook, Microsoft, Netflix, Twitter, Uber, and Yandex have invested tremendous resources in online controlled experiments (OCEs) to assess the impact of innovation on their customers and businesses. Running OCEs at scale has presented a host of challenges requiring solutions from many domains. In this article we review challenges that require new statistical methodologies to address them. In particular, we discuss the practice and culture of online experimentation, as well as its statistics literature, placing the current methodologies within their relevant statistical lineages and providing illustrative examples of OCE applications. Our goal is to raise academic statisticians’ awareness of these new research opportunities to increase collaboration between academia and the online industry.

KEYWORDS:

• A/B testing
• Literature review
• Online controlled experiments
• Randomized controlled trials
• Treatment effect estimation

1 Introduction

1.1 Background

It is estimated that in 2022, 5.16 billion people (64.4% of the world’s population) used the internet, each engaging with it on average 6.5 hr per day, and in aggregate spending over $5 trillion (USD) on consumer goods, travel and tourism, digital media, and health-related products and services (Kemp 2023). In 2023, e-commerce is predicted to account for 21% of all commerce, and by 2025 that number is expected to grow to nearly 25% (Keenan 2022). Given this scale of internet use, it is unsurprising that the optimization of online products and services is of great interest to online businesses and online components of traditional brick-and-mortar businesses.

Online controlled experiments (OCEs), digital versions of randomized controlled trials (RCTs) (Box, Hunter, and Hunter 2005), collect user-generated data to test and improve internet-based products and services (Kohavi and Longbotham 2023). Informally referred to as A/B tests, OCEs are an indispensable tool for major technology companies when it comes to maximizing revenue and optimizing the user experience (Luca and Bazerman 2021). Industry giants run hundreds of experiments on millions of users every day (Gupta et al. 2019), testing changes along multiple axes including: websites, services, and installed software; desktop and mobile devices; front- and back-end product features; personalization and recommendations; and monetization strategies. With OCEs, the causal impact of such changes—whether it be positive, negative, or zero—can be estimated. While this article focuses on experiments, we acknowledge that observational methods for causal inference are also relevant here, though their use is less prominent. We discuss them briefly in Section SM4 of the supplementary material.

While most positive changes are small, and improvement is incremental (Bojinov and Gupta 2022), results from OCEs can be potentially lucrative. Google’s famous “41 shades of blue” experiment is a classic example of an OCE that translated into a $200 million (USD) increase in annual revenue (Hern 2014); Amazon used insights from an OCE to move credit card offers from the homepage to the checkout page, resulting in tens of millions (USD) in profit annually (Kohavi and Thomke 2017); Bing deployed an A/B test for ad displays that resulted in $100 million (USD) of additional annual revenue in the United States alone (Kohavi, Tang, and Xu 2020). Even though such million-dollar ideas are relatively rare, the net gains from OCEs have been so profound that many organizations have completely overhauled their business models, with experimentation at the epicenter (Thomke 2020). For instance, Netflix attributes its membership growth from two countries to over 190 in the span of just 6 years to its adoption of online controlled experimentation (Urban, Sreenivasan, and Kannan 2016), and Duolingo’s 2022 Q2 shareholder letter attributes their growth to an “A/B test everything” mentality (Von Ahn 2022). The document even includes a description of their A/B testing process and several examples of how the product has evolved through experimentation.

Organizations that have accepted OCEs as standard practice generally adopt a so-called “culture of experimentation,” which is rooted in three tenets (Kohavi et al. 2013): (a) the organization wants to make data-driven decisions, (b) the organization is willing to invest in the people and infrastructure needed to run trustworthy experiments, and (c) the organization recognizes that it is poor at assessing the value of ideas. Generally, more than 50% of ideas fail to generate meaningful improvements (Kohavi, Tang, and Xu 2020). And in some domains, the failure rate of experiments (due to a combination of bad ideas or buggy implementation) is 90% or higher (Kohavi, Deng, and Vermeer 2022). Thus, carefully executed experiments provide a trustworthy, data-driven means to determine which ideas improve key metrics, which hurt, and which have no detectable impact, allowing the organization to invest in those that work, while pivoting to avoid the others. Within this culture, the attitude of “more, better, faster” is prevalent (Tang et al. 2010); organizations strive to increase the number of experiments so that all changes are properly evaluated; invalid experiments and harmful combinations of variants are straightforward to identify; and deployment, run time, and analysis occur within a relatively short period of time.

Compared to physical controlled experiments (in e.g., agriculture, manufacturing, pharmaceutical development), the cost incurred to design and run an OCE is low, even negligible for organizations with expertise in software development and statistics. Consequently, practitioners are able to run large numbers of experiments with potentially enormous sample sizes. In the case of large tech organizations, the combination of new features and modifications can result in billions of different versions of a given product (Kohavi et al. 2013). In these cases, hundreds of thousands of users are randomized concurrently to hundreds of experiments running simultaneously, so users may be in hundreds of experiments at a time (Gupta et al. 2019). Companies performing OCEs at this scale typically use experimentation platforms (software that is licensed or developed in-house) to automate the experimentation process, such as randomizing users, collecting data, managing concurrent experiments, and generating analysis reports (Tang et al. 2010; Fabijan et al. 2018; Ivaniuk 2020; Kohlmeier 2022). See Visser (2020) for a catalogue of in-house experimentation platforms developed by several prominent tech companies. Smaller companies tend to opt for third-party vendors that specialize in setting up, deploying, and analyzing OCEs. Several vendors were compared in Kohavi (2023) including (in alphabetical order): A/B Smartly, AB Tasty, EPPO, GrowthBook, Kameleoon, Optimizely, Split, Statsig, VWO, and Webtrends-Optimize. In all cases, this level of automation necessitates data quality checks like A/A tests and sample ratio mismatch (SRM) tests to establish trust in the experimentation platform. For further discussion of these practices and challenges, see Chapters 19 and 21 of Kohavi, Tang, and Xu (2020), and the introduction in Lindon and Malek (2020).

In this online setting, with the culture of testing as many ideas as possible, as quickly as possible, novel practical issues and modern challenges abound (see, e.g., Gupta et al. 2019; Bojinov and Gupta 2022; Quin et al. 2023 for nontechnical discussions, and Georgiev (2019) for a technical primer). The context in which OCEs operate departs markedly from the original applications for which traditional experiments were developed nearly a century ago; understanding this context is vital for developing relevant methodology for OCEs. For statisticians, online controlled experimentation provides a host of new opportunities for methodological and theoretical development. New approaches that fit the nuances of OCE applications are in high demand, with the majority of cutting-edge research spearheaded by those in industry. The purpose of this article, therefore, is to review the statistical methodology associated with OCEs, summarize its accompanying literature, and provide an overview of open statistical problems. We hope that increasing academic statisticians’ awareness of these research opportunities will help to bridge the gap between academia and the online industry.

1.2 The General Framework

Here we introduce the notation and key terms that will be used throughout this review and we describe the basic statistical framework for OCEs. It is useful to note that as a field, online experimentation has developed disparately across industries and domains, thus there are no unifying standards with respect to methodological approach and notation; even the term “controlled experiment” goes by different names depending on the organization: “flights” at Microsoft, “bucket tests” at Yahoo, “field experiments” at Facebook, and “1% experiments/click evaluations” at Google. The following notation largely draws from traditional RCT and causal inference literature, and is intended to help unify much of the OCE literature.

Let K be the number of variants (also known as buckets, arms, splits, and treatments) that compose the experiment. Ordinarily one of these variants is a control against which all other variants are compared. Unless explicitly stated, we shall assume for the rest of this review a standard treatment-versus-control setup, in which case K = 2. While multi-variant (K > 2) experiments exist in this space (colloquially referred to as “A/B/n tests”), we focus on the K = 2 “A/B test” for pedagogical reasons; even with K > 2 variants, determining which is optimal typically reduces to a pairwise comparison between each treatment and the control.

In A/B tests, n experimental units (e.g., users, cookies, sessions, etc.) are typically randomized in real time to one of the variants, and a response observation Y_i is collected for each $i=1,\dots ,n$. It is important to note that these response observations are typically themselves aggregates of more granular raw event data (Boucher et al. 2020). For instance, consider the response variable number of clicks per user which may be a count per user aggregated across sessions and/or pages. Interest then lies in optimizing some metric, which is a numerical summary of the response. Extending the previous example, interest may lie in maximizing the average number of clicks per user. Such metrics are often, but not always, averages. In some contexts, quantile or double-average metrics may be more suitable. We discuss such applications in more detail in Section SM4 of the supplementary material.

For simplicity of exposition, we have described a situation with one metric and hence one response variable. However, in practice there may be hundreds (even thousands) of metrics computed, many of which are used for debugging, some of which may be organizational guardrail metrics that the experimenters wish to avoid negatively impacting, and a small number of which compose the overall evaluation criterion (OEC) which is to be optimized. In general, defining and selecting metrics (as well as their corresponding randomization and analysis units) are key components of OCEs and we direct the reader to Kohavi et al. (2009), Deng and Shi (2016), Dmitriev et al. (2017), and Kohavi, Tang, and Xu (2020) for further discussion.

When the metric is an average, the primary goal of the experiment is to estimate the average treatment effect (ATE); the difference between the average outcome when the treatment is applied globally and when the control is applied globally. Within the potential outcomes framework (Neyman 1923; Rubin 1974), $Y_i(0)$ represents unit i’s response in the hypothetical scenario where i receives the control, and $Y_i(1)$ is the potential response when unit i receives the treatment. Letting W_i denote the binary treatment indicator for unit i, and given a particular treatment assignment to all experimental units $\mathbf{W}=(W_1,\dots ,W_n)\prime$, the expected outcome is $\mathbb{E}[\frac{1}{n}{\sum }_{i=1}^n{Y}_i(W_i)]=\mu (\mathbf{W})$, and the ATE is therefore given by (1) $$\begin{array}{c}\tau =\mu (\mathbf{1})-\mu (\mathbf{0})\hfill \\ =\frac{1}{n}\sum _{i=1}^n\mathbb{E}[Y_i(1)-Y_i(0)],\hfill \end{array}$$(1) where these expectations may be taken with respect to the random sampling or the random assignment, depending on whether inference is sample-based or design-based (Abadie et al. 2020). In reality, i can only be assigned to a single variant at a time, thus one cannot directly observe both $(Y_i(0),Y_i(1))$ and so the ATE is typically estimated with the difference-in-means estimator, (2) $$\widehat{\tau }=\frac{1}{n_1}\sum _{\{i:W_i=1\}}{Y}_i-\frac{1}{n_0}\sum _{\{i:W_i=0\}}{Y}_i,$$(2) where n₀ and n₁ are respectively the sizes of the control and treatment groups such that $n_0+n_1=n$. In practice it is also common to define the treatment effect as a relative percent, often referred to as lift, since it is easier to interpret and it is more stable (over experiment duration, for example).

Statistical significance is the most common mechanism by which a given treatment’s effectiveness is affirmed in an A/B test. Analyses of A/B tests are therefore most often carried out via two-sample hypothesis tests for τ with standard test statistics of the form $\widehat{\tau }/{\widehat{\sigma }}_{\widehat{\tau }}$. Such analyses, and the designs that generate data for them, commonly assume that the response of each user does not depend on other users’ treatment assignments (the Stable Unit Treatment Value Assumption, or SUTVA). SUTVA is a reasonable assumption for many scenarios; however, in Section 6 we discuss OCE settings where the assumption is violated and alternative methodologies are necessary. In many scenarios, sample sizes are large enough to confidently exploit the central limit theorem, permitting the use of the standard normal null distribution. There are, however, scenarios in which only a fraction of the user base is experimented on and asymptotic normality cannot be assumed. Such scenarios are discussed in Section 2.2. Given the heavy reliance on p-values it is important to acknowledge that the reproducibility crisis stemming from the misuse of hypothesis tests also plagues OCEs; p-value misinterpretation and problematic practices regularly lead to increased false-positive rates (Berman and Van den Bulte 2021; Kohavi, Deng, and Vermeer 2022). This is an area of ongoing practical and methodological concern in many fields, including online experiments.

1.3 Roadmap

With this context and foundation laid, we now review the statistical research in this area and discuss the many open problems. The article proceeds as follows. Section 2 discusses techniques for improving experimental power—a critical issue despite the relatively large sample sizes in OCEs. Sections 3 and 4, respectively, present literature regarding the challenges of estimating heterogeneous and long-term treatment effects. Section 5 discusses the problem of optional stopping and how sequential testing methods have been adapted to run online experiments. All of these sections presume SUTVA holds; we summarize the literature that explores violations of this assumption in Section 6. We conclude the review with a call to action for further collaboration between academia and OCE practitioners in Section 7. Note that a supplementary material file accompanies the article in which we provide expanded coverage of certain topics from the main text, as well as a brief discussion of additional topics outside the scope of this review.

2 Sensitivity and Small Treatment Effects

Motivating Example: Suppose an e-commerce website observes that 5% of their visitors make a purchase and the average purchase is $25 with a standard deviation of $\sigma =$ $6 during a one-week experiment period. Therefore, on average, a visitor spends $1.25. Suppose also the company’s annual revenue is $20 million, and gains or losses of $1 million (5%) are material. If the company wishes to run an experiment and detect a 5% change in revenue (i.e., $\delta =1.25\times 0.05$) with 80% power at a 5% significance level, a rough sample size calculation indicates they need $n_0=n_1=16{\sigma }^2/{\delta }^2=(16\times 6^2)/{(1.25\times 0.05)}^2=147,456$ users per variant. This is reasonable for a small startup. However, suppose now that the company’s annual revenue is $50 billion, with gains or losses of $10 million (0.02%) of interest. An experiment designed to detect a 0.02% change in revenue (i.e., $\delta =1.25\times 0.0002$) requires $n_0=n_1=16{\sigma }^2/{\delta }^2=(16\times 6^2)/{(1.25\times 0.0002)}^2=9.2$ billion users per variant, that is, 18.4 billion users in a single week. The human population on Earth is about 8 billion at the time of writing, so it is impossible for this company to detect changes that would gain or lose them $10 million per year.

Many leading organizations at the forefront of online controlled experimentation have user populations numbering in the hundreds of millions, if not billions. However, the sentiment that OCEs do not suffer from inadequate sample sizes is misconceived (Tang et al. 2010). Given the fundamental relationship between sample size and an experiment’s ability to detect true, nonzero treatment effects (namely its power), a key challenge facing even the largest of organizations is designing adequately powered experiments. A naive solution would be to simply extend the experiment’s duration, thereby increasing the number of users. However, as we elaborate upon in Section 4, this practice is ill-advised. Instead, it is better practice to employ a tactic that is tailored to the reason for insufficient power, which is generally one of three causes.

First, the treatment impacts the entire user population and the effect is roughly homogeneous, but very small in magnitude. As illustrated in the opening examples, even a fraction of a percent-change can translate to millions of dollars in revenue. We discuss the literature around this issue in Section 2.1. Second, many experiments test features that do not affect all users, attenuating the average treatment effect (see Section 2.2). Third, the treatment effects on known subpopulations are of interest, where sample sizes are smaller by definition (we defer this discussion to Section 3). In general, research regarding improving experimental power for OCEs tends to focus on boosting sensitivity, either by directly reducing the variance of Y_i or by reducing the variance of estimators for τ. The aforementioned subsections provide an overview of common methodology in this area. While specific methods of combating inadequate power are reviewed in this section, we encourage the reader to keep in mind that the issue of adequate power applies to all the challenges subsequently discussed in this review.

2.1 Transforming Y, Method of Control Variates, and Stratified Sampling

To improve sensitivity, a common approach is to transform Y into $Y^{*}$ of lower variance which, all else being equal, translates to a lower variance estimator of τ. In online experiments there can be dozens, even hundreds of metrics of potential interest, many with different properties that make it all but impossible to identify a “one size fits all” transformation. Much work has been devoted to documenting metric behavior and discussing techniques for metric definition. Kohavi et al. (2014) describe several examples of nonintuitive metric behavior and other peculiarities, illustrating the benefits of identifying skewed metrics and capping (truncating) them to improve sensitivity. Other transformations for improving the sensitivity of Y include binarizing count metrics and revenue. Deng and Shi (2016) define directionality (consistent behavior in one direction for positive treatment effects and in the opposite direction for negative effects) as an important feature when choosing metrics, suggesting that one should leverage prior experiments to compile a corpus of good metrics and to evaluate sensitivity and directionality with Bayesian priors. Deng and Shi (2016) also propose aggregating metrics in the form of a weighted linear combination, which is adopted and expanded upon by Kharitonov, Drutsa, and Serdyukov (2017). They frame finding sensitive combinations of metrics as a machine learning problem, incorporating both labeled and unlabeled data from past experiments. In Drutsa, Gusev, and Serdyukov (2015), features are extracted from data while the experiment is running and used to forecast metrics over a hypothetical post-experiment period. The authors also note their methodology may be applied to long-term effect estimation using statistical surrogacy, which we further discuss in Section 4.

In addition to transformations of Y, a popular approach is to define an efficient, mean-zero augmented estimator of τ using the method of control variates (Courthoud 2022; Sexauer 2022; Sharma 2022). Briefly, this method assumes, in addition to iid ${\left\{Y_i\right\}}_{i=1}^n$, the availability of independent observations of a covariate, ${\left\{X_i\right\}}_{i=1}^n$, such that $\mathbb{E}[X_i]={\mu }_x$. Often, these covariate measurements are collected from prior logs or experiments. Let $Y_i^{*}=Y_i-\theta (X_i-{\mu }_x)$, then $$\text{var}(Y_i^{*})=\text{var}(Y_i)+{\theta }^2\text{var}(X_i)-2\theta \text{cov}(Y_i,X_i)$$is minimized with respect to θ using the OLS solution $\text{cov}(Y_i,X_i)/\text{var}(X_i)$. Putting this together in the context of sample means gives $$\text{var}({\overline{Y}}^{*})=(1-{\rho }^2)\text{var}(\overline{Y})\le \text{var}(\overline{Y}),$$ where $\rho =\text{corr}(Y_i,X_i)$. Thus, an ATE estimator that uses the difference in treatment and control means of $Y_i^{*}$ tends to have lower variance than the traditional $\widehat{\tau }$, particularly when X_i is strongly correlated with Y_i. For OCEs, this technique is referred to as CUPED (Controlled experiments Utilizing Pre-Experiment Data) and was first proposed by Deng et al. (2013). The authors empirically demonstrate that an effective covariate choice is the same variable Y_i but collected during a pre-experiment period ($X_i\equiv Y_i^{\text{pre}}$). Such a choice can drastically increase sensitivity and thereby reduce time to statistical significance in evaluating $H_1:\tau \ne 0$. The authors also demonstrate that μ_x need not be known when X_i is uncorrelated with W_i and they also emphasize that despite resembling ANCOVA, CUPED does not require any linear model assumptions and can be treated as efficiency augmentation as in semi-parametric estimation (Tsiatis 2006). Consequently, CUPED has become a standard tool for many practitioners, although it is important to note that it can only be applied to users for which prior information exists (Drutsa, Ufliand, and Gusev 2015; Jackson 2018; Gupta et al. 2019; Hopkins 2020; Sharma 2021).

A key open question with respect to CUPED applications concerns the situation when the covariate alone is not sufficiently correlated with the response. An approach that shows promise employs synthetic controls, where one identifies a similar population without treatment exposure to use as covariates for modeling Y (Zhang et al. 2021). Another technique is to take advantage of a phenomenon that occurs in online experiments known as “triggering” (Deng et al. 2023), which we further discuss in Section 2.2. Further research with respect to the interplay between CUPED and other standard variance reduction techniques is also of interest. Xie and Aurisset (2016) apply CUPED to large-scale A/B tests for a subscription streaming service, and Liou and Taylor (2020) compare CUPED against variance-weighted estimators on a social media platform, finding that an aggregation of the two methods outperformed either individually. Deng et al. (2013) note that CUPED also permits nonlinear adjustments to the response variable. Following this, Poyarkov et al. (2016) develop an approach that assumes each user has a response Y and a set of features $\mathbf{F}\in {\mathbb{R}}^p$ independent of treatment assignment. Let $Y=f(\mathbf{F})$, where f is an unknown, nonparametric function that is estimated with machine learning. Following the general idea of control variates, the covariate is chosen to be the predicted outcomes of $\widehat{f}$. Poyarkov et al. (2016) then use $Y^{*}=Y-\widehat{f}(\mathbf{F})$ as the primary metric for estimating the ATE, noting an increase in sensitivity compared to traditional A/B tests.

Closely related to the method of control variates/CUPED is stratified sampling. We discuss these connections as well as the use and drawbacks of stratified sampling in more detail in Section SM1 of the supplementary material.

2.2 Triggered Analysis

Motivating Example: Suppose engineers are testing a change made on an e-commerce website’s checkout page. Users in the experiment who never interact with this checkout page are not impacted by the experiment and so their treatment effect is zero. Many such users will increase noise and dilute the treatment effect. Sensitivity may be increased by analyzing only the users who could have been impacted by the experiment; those that were triggered into the analysis. Although this reduces sample size, the treatment effect among the triggered users is undiluted and therefore higher and easier to detect.

Triggered analysis broadly refers to an OCE analysis where one only considers users who have the potential of being impacted by an experiment, excluding those who would not be affected by the proposed variant (Deng et al. 2023; Kohavi et al. 2009; Kohavi 2012; Xu, Duan, and Huang 2018). Users are said to have triggered the experiment when they exhibit behavior that results in direct exposure to their assigned variant. Such users could be assigned to a variant either early (e.g., when first visiting the web site) or when they exhibit some type of behavior that triggers the experiment (e.g., when reaching the checkout page, which impacts the variant they receive). Key analysis challenges include: (a) generalizing the results from the triggered users to a broader population, and (b) reducing the variance of τ estimators to offset the smaller sample sizes that result from triggering. For an in-depth discussion of triggering case-studies, including the example above, see Chapter 20 of Kohavi, Tang, and Xu (2020). We will provide a brief description and an overview of research in this area.

Let Ω be the overall user population and $\Theta \subset \Omega$ the population of users who could be affected by the treatment. A given user is determined to belong to Θ via techniques such as conditional checks or counterfactual logging (Kohavi, Tang, and Xu 2020; Deng et al. 2023). If Θ comprises only a modest fraction of Ω, (i.e., $\left|\Theta \right|/\left|\Omega \right|\le 0.2$, for instance), an experiment that samples data from the entire population could be severely under-powered, particularly when effect sizes are small (Kohavi et al. 2009). To mitigate this issue, practitioners focus analysis only on triggered users. The difference-in-means estimator ${\widehat{\tau }}_{\Theta }$ is an unbiased estimator for the ATE of the triggered population, ${\tau }_{\Theta }$, under standard assumptions. However, ${\tau }_{\Theta }$ is typically larger than the population-level ${\tau }_{\Omega }$ and the corresponding estimator generally has greater variance. The process of estimating ${\tau }_{\Omega }$ from ${\widehat{\tau }}_{\Theta }$ is referred to as estimating the diluted treatment effect. This allows the business to estimate the global impact of launching a new feature to all users (some of whom will be directly impacted, and some of whom will not).

Most triggered analyses fall under the following framework. Assume a (not necessarily random) sample of N units, n of which are triggered. Each user i interacts with the website on multiple separate events. During each event, i may or may not trigger (e.g., i may interact with the checkout page during one event, but not the other). The most common analysis technique is the user-trigger analysis, which incorporates all events beginning with the first event where i triggered. Such analyses are quite popular as they do not require any assumptions regarding the treatment effect, and are amenable to common user-level metrics. Chen, Liu, and Xu (2018) use the user-trigger framework to illustrate the benefits of triggered analyses in terms of power gains and variance reduction, as well as to highlight the types of biases that may occur under such approaches. The session-trigger analysis is another approach that groups events into “sessions” and only keeps sessions that contain trigger events for analysis. While Deng and Hu (2015) note that estimates from session-triggered analyses do tend to have lower variance than user-trigger analyses, the treatment effect must be zero in the nontriggered sessions in order for this approach to be valid. While perhaps true in some cases, generally this assumption is difficult to verify for most applications.

One approach for estimating the diluted treatment effect is to derive ${\tau }_{\Omega }$ in terms of ${\tau }_{\Theta }$, producing so-called “diluted formulas.” For additive metrics, $Y_i=Y_{i,t}+Y_{i,u}$, where $Y_{i,t}$ is an outcome when i is triggered and $Y_{i,u}$ for when i is untriggered, it can be shown that the diluted treatment effect is the average treatment effect on triggered users weighted by the proportion of triggered users, that is, ${\tau }_{\Omega }={\tau }_{\Theta }\times \frac{n}{N}$. Note that this expression only applies when, for $i\in \Theta$, there is no treatment effect on the sessions where i is untriggered, that is, $Y_{i,u}(1)-Y_{i,u}(0)=0$. In other words, this expression is only for valid session-trigger analyses. With ratio metrics, $Y_i=a_i/b_i$, if there is no treatment effect for the denominator term, $b_i=b_i(1)=b_i(0)$, and the rate at which users are triggered into the experiment is independent of ${\tau }_{\Theta }$, then the diluted formula is ${\tau }_{\Omega }={\tau }_{\Theta }\times \frac{n}{N}\times \overline{r}$, where $\overline{r}$ is the average trigger rate as a function of b_i. Further details for these derivations may be found in Deng and Hu (2015), but it is important to note that these formulas only apply when the treatment effect defined in (1) is of interest. When a relative effect (i.e., lift) is of interest, one must perform relative dilution where weighting is not by the proportion of triggered users, but by their contribution to the metric. While these formulas are certainly helpful in illustrating the connection between ${\tau }_{\Theta }$ and ${\tau }_{\Omega }$, they are restrictive because their underlying assumptions are not necessarily realistic and closed-form expressions only exist for special cases. As noted by Deng and Hu (2015), the trigger rates are rarely independent of the triggered treatment effect; users who visit a website frequently will have higher trigger rates and tend to have a larger treatment effect than less-frequent users (see Wang et al. 2019).

The above formulas provide a solution for estimating ${\tau }_{\Omega }$ but they do not address the issue of low power that typically afflicts triggered analyses. Deng and Hu (2015) and Deng et al. (2023) simultaneously address both issues by formalizing the connection between all diluted formulas and variance reduction. Under the assumption that there is no treatment effect when users are untriggered, Deng and Hu (2015) apply the CUPED framework (Section 2.1) by augmenting ${\widehat{\tau }}_{\Theta }$ with mean-zero data from the trigger complement group. The authors show that the resulting augmented estimator is unbiased for ${\tau }_{\Omega }$, can achieve appreciable variance reduction, and applies to metrics of any form. Deng et al. (2023) extend this application of CUPED to one-sided triggering, a type of one-sided noncompliance where only the triggering status of the treatment group is observed.

Compared to other challenges in OCEs, the literature for triggering is, at present, rather sparse. Consequently, there are still many areas open for further research. The discussed methodologies for estimating the diluted treatment effect each depend on assumptions that may be too restrictive in certain applications. An additional challenge concerns bias of standard ATE estimators induced by triggering. Chen, Liu, and Xu (2018) identify a special type of bias that occurs when a user triggers on day d, but not day d + 1. Other types of bias, as well as the questions of how to define the randomization unit (user, session, or webpage) and how and when to aggregate data into sessions, remain open for exploration. Recent work in Deng et al. (2023) also suggests that exploring noncompliance and other similar concepts from the causal inference literature (such as principle stratification) with respect to triggering may be an area for future development.

3 Heterogeneous Treatment Effects

Motivating Example: Suppose an online ad provider wishes to determine the impact of changing from static textual ads to short video ads on website traffic. For the treatment group, website traffic appears to have increased uniformly except among Safari users. Consequently, the ad team wishes to estimate the treatment effect at the browser level. Likewise, after observing an improvement in user engagement metrics, the ad team may want to perform a post-hoc analysis to determine if this increase is roughly the same for all users or perhaps concentrated within certain user segments (such as those defined by market/country, user activity level, device/platform type, and time).

Treatment effects on subgroups differing from the population-level ATE are known as heterogeneous treatment effects (HTE) and are commonly of interest in OCEs. Identifying and interpreting such heterogeneity is vitally important for business applications. For example, practitioners may be interested in estimating the treatment effect for different devices or browser types, or for users of different ages, or users living in different parts of the world. Identifying and estimating HTEs is also of concern for those wishing to develop personalized experiences, or to detect bugs, or interactions with other experiments. Three key challenges are associated with estimating HTEs: (a) small treatment effects (see Section 2) often make online studies under-powered, resulting in high false negative rates for subgroup effects; (b) testing for HTEs tends to risk inflated Type I error rates due to multiple comparisons; (c) in cases where users are not randomized to the subgroups under comparison, the usual tension between correlation and causation manifests. Below we review existing methodologies that are commonly used in the context of OCEs to address these problems.

Heterogeneous treatment effects have a rich history in statistical theory and application (Robinson 1988; Zhao et al. 2012; Imai and Ratkovic 2013; Athey and Imbens 2016; Wager and Athey 2018; Tran and Zheleva 2019). In this review, we focus on the intersection of this literature and OCEs. Assume each unit i has a pair of potential outcomes $\{Y_i(1),Y_i(0)\}$ and a vector of pretreatment covariates $X_i$, with $e(x)=Pr(W_i=1|X_i=x)$ being the probability that a user is treated given a particular value of the covariate. For randomized studies where causality may be inferred, $e(X_i)$ is known and technically independent of $X_i$; however, when HTE analysis is under an observational setting, $e(X_i)$ is typically unknown. Most of the literature regarding HTEs employs the following assumptions: (1) SUTVA and (2) unconfoundedness, meaning that the response is independent of the treatment assignment W_i conditional on the covariate, $\{Y_i(1),Y_i(0)\}\perp \text{}\perp W_i|X_i$. The main goal is to estimate the conditional average treatment effect $$(\mathit{CATE}),\tau (x)=\mathbb{E}[Y_i(1)-Y_i(0)|X_i=x].$$

Another key challenge is to detect exactly for which specific levels of the covariate $\tau (x)$ differs from τ and, given several covariates, identifying which X’s are the source of heterogeneity.

Interpretation is crucial in the online industry, thus, a popular approach is to assume a linear mapping from Y_i to $(W_i,X_i)$ from which main and interaction effects may be estimated. Unsurprisingly, the relationship between Y_i and X_i is often highly complex, thus a common method is to use the semi-parametric model from Robinson (1988), $Y_i=\tau (X_i)W_i+g(X_i)+{\epsilon }_i$, which makes no assumptions about the forms of $\tau (X_i)$ and $g(X_i)$. Under unconfoundedness, one may write $Y_i-m(X_i)=\tau (X_i)(W_i-e(X_i))+{\epsilon }_i$, where $m(X_i)=\mathbb{E}[Y_i|X_i]$ and $e(X_i)$ are unknown. The ${\mathcal{l}}_2$ loss function is used to estimate the heterogeneous treatment effects, resulting in the estimate $$\widehat{\tau }(X)=\underset{\tau \prime }{\text{argmin}} \frac{1}{n}\sum _{i=1}^n{\left\{Y_i-m(X_i)-\tau \prime (X_i)\left[W_i-e(X_i)\right]\right\}}^2.$$

Thus HTE estimation is a ripe target for machine learning methods. Researchers have approached this problem using a technique called “Double Machine Learning” (DML) (Chernozhukov et al. 2017). Briefly, this technique models m(X) and e(X) as nuisance parameters, estimating them with nonparametric regression on a hold-out sample set. The CATE may then be estimated on the remaining sample set using a variety of machine learning methodologies. Chernozhukov et al. (2017) demonstrate that the above squared error loss is Neyman orthogonal to m(X) and e(X), which, along with sample splitting, ensures unbiasedness of $\widehat{\tau }(X)$ and enforces parsimonious modeling of Y with respect to the nuisance parameters. Syrgkanis et al. (2019) extend DML for estimating heterogeneous treatment effects when the covariates are hidden. Such situations arise in online experiments when users do not comply with a treatment due to unobserved factors. By modeling Y with instrumental variables, Syrgkanis et al. (2019) estimate the HTEs using a doubly-robust, fully convex loss function that is minimized with an algorithm that builds on the DML technique. To avoid the challenge of directly estimating the HTE, Peysakhovich and Lada (2016) use historic, user-level data to learn individual effect estimates conditional on the covariates that correlate with the true treatment effect. Practitioners at Netflix also used DML to understand the localized impact on viewership from subbed and dubbed movies (Lan et al. 2022).

Other popular machine learning approaches for estimating $\tau (X)$ are regression trees and random forests. Following the DML approach, for instance, one may use trees to identify meaningful segments of a continuous or categorical variable, and then model $\tau (X)$ with partially linear regression. In an adaptation of the classical CART algorithm, Athey and Imbens (2016) build modified regression trees to partition the data into subgroups corresponding to different magnitudes of the treatment effect, thus each terminal leaf produces an estimate for $\tau (x)$, rather than the traditional estimate of $\mathbb{E}[Y|X_i=x]$. To correct for over-fitting, an additional split of the training data into nonoverlapping sub-partitions for each leaf is used. Naturally, this method can be extended to random forests to create a causal forest for estimating the HTE (Wager and Athey 2018). While casual trees and forests do not require linearity of the treatment effects, and perhaps are conceptually more intuitive than DML, they are somewhat lacking in terms of interpretability compared to the effect estimates from DML and other similar methods. A further disadvantage is that the additional training split reduces the sample size for an application that may already suffer low power.

In addition to estimating $\tau (X)$, identifying which covariates or levels of covariates contribute to treatment heterogeneity is of great practical concern (Sepehri and DiCiccio 2020; McFowland III et al. 2021). Obtaining a parsimonious model of Y is critical in such situations, as there are typically a large number of covariates from which to choose and strong statistical significance is required for detecting HTEs. This challenge is simultaneously a variable selection and multiple testing problem. Xie, Chen, and Shi (2018) assume an experimental design setup where $e(X)=Pr(W_i=1)$, using this value to transform Y_i into $Y_i^{*}$ such that $\mathbb{E}[Y_i^{*}|X]=\tau (X)$, which is estimated with the standard difference-in-means $\widehat{\tau }$. Using $Y^{*}-\widehat{\tau }$ as the response variable, the authors perform Lasso regression in conjunction with the “knockoff” variable selection defined by Barber and Candès (2015) to select heterogeneous covariates while controlling false discovery rates (FDRs). They also demonstrate how to use the Benjamini-Hochberg correction to identify levels within these covariates where HTEs occur. Deng et al. (2016b) also use variable selection when clusters of covariates are of interest, such as device grouped by brand name. They employ a linear model with first order effect and second order interaction terms and enforce sparsity using total variation regularization, a technique similar to Fused Lasso (Petersen, Witten, and Simon 2016).

Given the wide array of scenarios under which HTEs occur in online experiments, there are still many situations where the methods discussed above may not be appropriate. Much of the literature in this review make strong model assumptions that are difficult to verify in practice. For example, the SUTVA requirement that units in a given variant receive the same level of treatment may not be satisfied if users have different levels of engagement (Imbens and Rubin 2015); highly engaged users will typically experience a higher “dose” of their treatment (relative to light users) due to repeated exposure over the experimental period. As such, the vast literature on dose-response studies Ruberg (1995a, 1995b) seems pertinent here. Additionally, the low power due to small effect sizes makes multiple testing quite challenging. Simulations regarding the approach for controlling FDR in Xie, Chen, and Shi (2018) showed that the knockoff method may be too conservative when faced with small effect sizes, and Deng et al. (2016b) reported difficulties regarding high false positive rates. For more open challenges regarding HTE estimation, we encourage the interested reader to consult Gupta et al. (2019), Kohavi, Tang, and Xu (2020), and Bojinov and Gupta (2022).

4 Long-Term Effects

Motivating Example: At Bing, researchers hypothesized that generating large numbers of advertisements should have a positive effect on revenue, but may hurt user engagement in the long-term. To test this, the researchers exposed users to varying ad loads, noting a significant difference in engagement metrics for users exposed to a high ad load versus a low one. It was proposed that one may estimate the long-term effect by performing a post-hoc analysis some time after the experiment. Unfortunately, the post-hoc differences between high-load and low-load users could not be solely attributed to treatment assignment—many users quickly abandoned Bing as a result of too many ads, biasing results toward the users who remained (Dmitriev et al.2016).

Practitioners are often interested in understanding the treatment effect not just during the experiment, but months, even years after the experiment concludes. In many online experiments, the short-term treatment effect observed during and immediately after the experiment is not necessarily the same as the long-term effect. For instance, click-bait advertising has a positive short-term effect on click-through-rates, but a negative long-term effect on user retention and revenue (Kohavi et al. 2012). More generally, novelty and primacy effects are of concern. A novelty effect exists when a novel change is initially intriguing, leading to increased engagement, but that diminishes over time. A primacy effect on the other hand exists when the initial reaction to a change is not positive, but over time as users get used to the change their engagement increases (McFarland 2012; Sadeghi et al. 2022). In both cases, the nature of the treatment effect may change over time as users learn. The treatment effect itself may also change dynamically over time independent of user learning. As such, the ATE τ should not be viewed as a constant with respect to time (t), it should more appropriately be regarded as a function of it: $\tau (t)$.

Current OCE literature regarding long-term effect estimation is highly context-specific. At the time of writing this review, it is difficult to pinpoint a single statistical lineage of methodologies for this area (unlike with heterogeneous treatment effects, for example). We begin by introducing several distinct approaches that draw from a variety of statistical fields, and then finish with discussion of one area in particular that shows promise in providing a statistical framework for modeling and estimating long-term effects in online settings. For more industry-specific examples of the challenges concerning long-term effects, see Gupta et al. (2019) and Bojinov and Gupta (2022).

A straightforward way to assess long-term effects is to simply run the experiment longer and ensure that the appropriate metrics for capturing long-term behavior are observed. However, much of the literature written by practitioners of OCEs has been devoted to describing the pitfalls associated with running long-term controlled experiments specifically for estimating long-term effects (Kohavi et al. 2009, 2012; Dmitriev et al. 2016; Gupta et al. 2019; Kohavi, Tang, and Xu 2020). Besides increased cost, several other external factors often make long-term experiments unappealing. For instance, when browser cookies are used to identify users, long-term experiments risk losing upwards of 75% of users as a result of cookie churn and are rendered invalid as a result (Dmitriev et al. 2016). These users may also reenter the experiment unbeknownst to the experimenters and receive both the treatment and control experiences. This type of contamination can also happen if users access the product or service on multiple devices, a problem that becomes more likely as the experiment’s duration increases. Additionally, the longer the experiment the more likely it is that multiple users (e.g., family members) will use the same device, obfuscating results. As such, in this section, we focus on techniques for estimating long-term treatment effects alternative to increasing experiment length.

Several approaches for estimating long-term effects intersect with other areas discussed in this review. In Wang et al. (2019), long-term effects are characterized as a form of bias due to heterogeneous treatment effects (Section 3). In this context, long-term effects manifest because heavy-users (frequent users of the product) tend to be included in experiments at higher rates than light-users, biasing the ATE particularly in the short-term. Here, the treatment effect is presumed to be different depending on whether user i is a heavy- or light-user. Under SUTVA and an assumed independence of outcomes from treatment assignment, the authors derive bias due to heavy-users in closed form, proposing a bias-adjusted jackknife estimator for the overall ATE. For a two-sided market where each experimental unit has a treatment history up to time t, Shi et al. (2020) leverage sequential testing (Section 5) and reinforcement learning to test for long-term treatment effects. Using data from a ride-sharing company, they demonstrate how their derived test statistic is able to detect long-term effects where regular two-sample t-tests fail. While the solutions from Wang et al. (2019) and Shi et al. (2020) are effective, they only target specific types of long-term effects, which limits their potential generalizability to other settings.

Another common solution is to define and measure short-term driver metrics that are causally linked to the long-term effect (Kohavi, Tang, and Xu 2020). Driver metrics allow practitioners to focus experiments on short-term goals while still taking into account the long-term effects (see Biddle (2019) for anecdotal examples). Hassan et al. (2013), define heuristics for modeling implicit indicators of customer satisfaction, noting that using query-based models instead of click-based models tend to serve as better proxies. Hohnhold, O’Brien, and Tang (2015) define models for how users “learn” to search or click for a product as a result of being exposed to a treatment, such as change in number of ads shown, using “learned click-through-rates” as a driver metric for estimating long-term effect on revenue. To estimate the effect on long-term revenue using short-term effects due to treatment, the authors model this as a linear function of short-term revenue and the estimated learned click-through-rates. The model has been successfully deployed by Google and is widely cited in the OCE literature (Deng, Lu, and Litz 2017; Gupta et al. 2019; Wang et al. 2019; Kohavi, Tang, and Xu 2020). A recent paper by Sadeghi et al. (2022) proposes an observational approach based on difference-in-differences to estimate user learning and hence the long-term treatment effect in contexts where novelty and primacy effects exist.

Methodology in this area tends to resemble recent works in the causal inference literature that also aim to address this challenge by combining short-term experimental data with long-term observational data. In Section SM2 of the supplementary material we elaborate on the use of surrogate outcomes (Prentice 1989; Begg and Leung 2000; Frangakis and Rubin 2002; Ensor et al. 2016) in this context.

5 Optional Stopping

Motivating Example: Suppose an online streaming service is altering a certain feature that positively correlates with subscription renewals. While an improvement to this feature could increase the rate of subscription renewals, a harmful change may have the opposite effect. It is in the service’s best interest to quickly abandon harmful or poorly performing variants, and identify those that perform well. Methods that support early termination without compromising overall statistical validity are desirable. A notable practice within this context is to “ramp up” the experiment by slowly exposing an increasing percentage of users to the treatment (Xu, Duan, and Huang 2018), and to integrate OCEs into a phased software deployment via “controlled rollout” (Xia et al. 2019).

Most OCEs are run in real-time, and it is not uncommon for estimates and confidence intervals associated with τ, and p-values associated with $H_0:\tau =0$, to be updated in near-real-time as the data are collected. Although a fixed horizon is typically determined based on development cycles (typically two weeks) and minimum sample size requirements (determined via power arguments), the near-real-time availability of results encourages a phenomenon known colloquially as “peeking,” whereby p-values are monitored continuously and the experiment is stopped as soon as a significant p-value is observed. While it is well known that this practice seriously inflates false positive rates (Johari et al. 2022a; Kohavi, Deng, and Vermeer 2022), there are nevertheless situations where having a mechanism for optional stopping is desirable. For example, it is extremely important to quickly detect and abort treatments that are negatively impacting the user experience (Lindon, Sanden, and Shirikian 2022). Thus, in situations like this, a methodology that permits near-real-time decision-making without inflating Type I error rates is invaluable. Unsurprisingly, design and analysis methods from the sequential testing body of literature are of relevance here. Below we describe the development and application of methods in this area to the OCE context.

As per the field of sequential testing, interest lies in assessing $H_0:\tau =0$ using sample size-dependent decision rules. Within this class of methods, Type I error is controlled at each current sample size n, which avoids the inflated risk of Type I error that is associated with preemptively stopping an experiment when the current p-value is statistically significant by chance. Such methods improve testing efficiency due to the lower sample size a sequential test will terminate at, on average, regardless of where the true treatment effect might lie. However, there is no free lunch. Existing methodology is not well-suited for all OCE applications, such as monitoring multiple metrics (e.g., the OEC and guardrails). Additionally, the reduced sample sizes guarantee under-powered HTE inference across user segments. Nevertheless, sequential testing methods have appeared in numerous OCE applications and studies (Kohavi et al. 2013; Kharitonov et al. 2015; Deng, Lu, and Chen 2016; Abhishek and Mannor 2017; Ju et al. 2019; Yu, Lu, and Song 2020; Shi et al. 2020; Johari et al. 2022a; Schultzberg and Ankargren 2023; Skotara 2023). The following section broadly introduces the method of sequential testing as it pertains to ongoing evaluation of the treatment effect(s) of interest in OCEs.

The majority of the OCE literature in sequential testing builds on the classic sequential probability ratio test (SPRT) developed by Wald (1945). Define constants $0<B<A$ where $B=\beta /(1-\alpha )$ and $A=(1-\beta )/\alpha$, and a simple hypothesis test $H_0:\theta ={\theta }_0$ versus $H_1:\theta ={\theta }_1$. The SPRT method proceeds as follows. For current sample size n, compute the likelihood ratio test statistic $${\Lambda }_n=\prod _{i=1}^n\frac{f(y_i|{\theta }_1)}{f(y_i|{\theta }_0)},$$where y_i are observations of iid data ${\left\{Y_i\right\}}_{i=1}^n\sim f(·|\theta )$. The rejection region divides the sample space into three mutually exclusive decision rules: (a) if ${\Lambda }_n>A$, reject H₀ and stop the test. (b) If ${\Lambda }_n<B$, fail to reject H₀ and stop the test. (c) If $B<{\Lambda }_n<A$, obtain another observation $Y_{n+1}$ and compute ${\Lambda }_{n+1}$. Although it seems like testing in this manner would permit the possibility of never drawing a conclusion about H₀ (i.e., $n\to \infty$), Wald (1947) proved that the SPRT will eventually terminate for finite n. SPRT does not require specifying n in advance, and requires on average about half the number of observations required for a uniformly most powerful Neyman-Pearson test for the same level of power (Wald 1945).

The first and perhaps most widely-known application of sequential testing in OCEs is a modified version of SPRT called the mixture sequential probability ratio test, or mSPRT (Pramanik, Johnson, and Bhattacharya 2021; Johari et al. 2022a). The mSPRT allows for a simple null hypothesis versus a composite alternative hypothesis $H_1:\theta \ne {\theta }_0$ by assuming a mixture distribution H with density $h(·)$ defined over the parameter space of all possible θ. The test statistic is therefore a mixture of the likelihood ratios, $${\Lambda }_n^H=\int \prod _{i=1}^n\frac{f(y_i|\theta )}{f(y_i|{\theta }_0)}h(\theta )d\theta .$$

The procedure rejects H₀ and ends if ${\Lambda }_n^H\ge {\alpha }^{-1}$. Johari et al. (2022a) use the mSPRT to define “always valid p-values”, which are computed iteratively such that $p_0=1;p_n=\min \{p_{n-1},{({\Lambda }_n^H)}^{-1}\}$. Thus, practitioners may stop an experiment at any time while still controlling Type I error. The always valid p-values and their confidence interval counterparts are currently deployed by Optimizely, a widely-used third-party vendor for OCEs (Pekelis 2015). With the vast quantity of experiments that this company has facilitated, they are able to leverage prior data for estimating the mixture distribution H. However, Johari et al. (2022a) derive their optimality conditions for mSPRT only for data that comes from the exponential family of distributions, which does not include distributions for the ratio metrics that are popular in industry. Another limitation lies in how the likelihood ratios for a two-sample hypothesis test are defined. While the authors assume a standard independent, two-sample stream of data, they impose an additional restriction by arbitrarily pairing observations, which suggests a matched pairs design. This allows for defining a tractable $f(y_i|\theta )$, but there is no practical reason for observations to be paired, and no practical guidance given for how to perform the pairing. Additionally, the assumption that observations arise independently may be violated when a unit generates multiple observations as they interact repeatedly with the experiment over time. Moreover, although the Type I error rate is satisfactorily controlled (when all assumption are met), unbiased estimation of the treatment effect is still a concern. Methodology that relaxes these assumptions and yields unbiased estimates is therefore valuable.

The well-publicized usage of mSPRT has inspired several related works in the literature. Abhishek and Mannor (2017) account for the situation where $f(·|\theta )$ is unknown by creating a bootstrap algorithm to approximate ${\Lambda }_n^H$. While the algorithm also requires a prior distribution to approximate H, this method still allows practitioners to use mSPRT for commonly used online metrics that are otherwise difficult to model. The work in Lindon and Malek (2020) extends mSPRT to multinomial count data, which includes an application for conducting SRM tests sequentially, in near-real-time. Yu, Lu, and Song (2020) also extend mSPRT to the multiple testing scenario to test for heterogeneous treatment effects, using always valid p-values to allow for continuous monitoring. Xu, Duan, and Huang (2018) use a technique similar to mSPRT, called a generalized sequential probability ratio test (GSPRT), to determine the risk of exposing more users to a new variant. Briefly, the GSPRT uses the supremums of the likelihoods in Λ_n and can be shown to require smaller sample sizes on average than mSPRT (Chan and Lai 2005). Xu, Duan, and Huang (2018) use a prior-weighted GSPRT to provide a rigorous statistical framework dubbed “speed, quality, and risk” (SQR) for the practice of ramping up, gradually introducing users to a new variant in order to mitigate the fallout associated with exposing them to potentially negative variants (for a high-level discussion of SQR, see Chapter 15 of Kohavi, Tang, and Xu 2020). An alternative to frequentist sequential testing is also explored by Deng, Lu, and Chen (2016), where the authors use Bayesian hypothesis testing as the foundation. Bayesian methods for OCEs are briefly discussed in Section SM4 of the supplementary material.

Finally, we acknowledge that the sequential testing methods discussed here are fully sequential. However, group sequential methods commonly used in adaptive clinical trials (Pocock 1977; O’Brien and Fleming 1979; Lan and DeMets 1983; Robertson et al. 2023) are rapidly gaining in popularity in the context of OCEs. Georgiev (2022), Skotara (2023), and Schultzberg and Ankargren (2023) describe the use of these methods in this context, and the value they provide with respect to the speed of decision making (i.e., increased power) and false positive control. The adaptation of such methods to tailor them for use with OCEs seems like a nascent though fruitful line of research.

6 Interference

Motivating Example 1: Suppose LinkedIn plans to test the impact of a new feature for their messaging service, with the objective to increase total messages sent. Using balanced randomization, given that user i is exposed to the new feature, there is approximately a 50% chance said user’s friend j is randomized to the old service. Under this scenario and if the new feature indeed increases messages sent, it is likely that friend j will also send more messages in response to i, despite j belonging to the old service. Thus, the overall impact of the new messaging feature on total messages sent is confounded by the network interference between treatment and control groups, biasing standard estimators for the ATE (Saint-Jacques 2019).

Motivating Example 2: Suppose Lyft is experimenting with a new version of the pricing algorithm that results in treated passengers booking more rides. However, since the number of available drivers is finite, increased ride bookings in the treatment group necessarily reduces the supply of drivers and hence the number of possible rides for the control group (Chamandy2016). This in turn biases naive treatment effect estimates that compare ride bookings in the two groups.

Recall that SUTVA requires that the potential outcome $Y_i(W_i)$ for unit i remain the same regardless of the treatment assignments and outcomes of the other experimental units. However, in certain OCE applications (e.g., social networks and online marketplaces), SUTVA may be violated when the treatments interfere with each other. Such a SUTVA violation is referred to as interference, spillover, or leakage. Interference was illustrated in both motivating examples above; in each case a unit’s outcome depended not only on its own treatment assignment, but also on the treatment assignments and outcomes of other units in the experiment. The two examples typify two different forms of interference: the first, network interference, arises when the units are connected to one another through a network, such as a social network like LinkedIn (Saint-Jacques et al. 2019) or Facebook (Eckles, Karrer, and Ugander 2014). The second form of interference, marketplace interference, arises when units compete for shared resources in two-sided marketplaces such as Lyft (Chamandy 2016), Ebay (Blake and Coey 2014), or Airbnb (Holtz et al. 2020), and three-sided marketplaces like DoorDash (Feng and Bauman 2022). Note that Kohavi, Tang, and Xu (2020) defines these interference mechanisms as resulting (respectively) from “direct” and “indirect” connections among the units. Bojinov and Gupta (2022) alternatively define partial and arbitrary interference in addition to marketplace interference. Whether the interference mechanism is partial or arbitrary depends on whether a unit is influenced by some or all of the other units in the experiment. Kohavi, Tang, and Xu (2020) describe another type of interference whereby a malfunctioning treatment causes a crash that impacts both treatment and control users.

As described in Section 1.2, estimates of the average treatment effect seek to quantify the difference in outcomes when all units are treated versus when all units are controlled. In standard settings, a subset of units randomized to treatment and another subset randomized to control serve as an adequate proxy for the unobserved counterfactuals. However, when the treatment and control groups interfere with each other, traditional randomization no longer adequately approximates the counterfactuals and hence standard difference-in-means estimators are no longer unbiased. A rich literature has recently been developed that carefully considers both the design and analysis of OCEs in the presence of interference. Of particular relevance are experimental designs that reduce the amount of interference, and analysis methods that provide unbiased treatment effect estimates in the presence of interference. We provide a brief overview of that literature below. As will become apparent, many ideas from the clinical RCT literature are relevant here (e.g., cluster randomized trials and crossover designs).

In network A/B tests, cluster-based randomization methods have gained widespread attention (Ugander et al. 2013; Eckles, Karrer, and Ugander 2014; Gui et al. 2015; Saveski et al. 2017; Yoon 2018; Zhou et al. 2020; Karrer et al. 2021). Such methods first involve partitioning the network into mostly disjoint clusters, commonly via community detection algorithms. Such a partitioning yields groups of nodes with much greater intra-cluster connectivity than inter-cluster connectivity. Randomization is then performed at the cluster level, where all units in a cluster receive the same treatment assignment. In doing so, units will (for the most part) have the same treatment assignment as the units nearest—and hence most likely to influence—them. This limits the opportunity for interference and therefore better mimics the all-treated and all-controlled counterfactuals. However, with the clusters being the randomization unit (instead of the individual users), the effective sample size and hence power is dramatically reduced. Saint-Jacques et al. (2019) therefore propose the use of many, smaller ego-clusters. These clusters are defined by a single user (the ego) and a subset of its direct neighbors (the alters). Ego-cluster-based randomization does not pay as significant a penalty in terms of power, since there are many more ego-clusters than in a traditional cluster-based design. Moreover, other treatment assignment schemes in which the ego is treated differently from the alters may be used to estimate the network interference. In Section SM3 of the supplementary material we elaborate on these and other methods developed for the design and analysis of network A/B tests.

The spirit of cluster-based randomization—that is, to treat units likely to influence each other in the same way—is at the heart of methodologies intended to mitigate other forms of interference as well. For instance, in settings with marketplace interference, switchback experiments are commonly used to sequentially alternate units between treatment and control over time (Bojinov, Simchi-Levi, and Zhao 2022). In doing so, at any given time period all users have the same treatment assignment and therefore cannot exert influence on each other. And then through repeated exposure to treatment and control over time, the ATE can be estimated. Such experiments suffer from temporal carryovers, but these can be mitigated with “burn-in” periods analogous to washout periods in clinical trials (Hu and Wager 2022). Optimal design strategies have also been developed to address carryover effects (Bojinov, Simchi-Levi, and Zhao 2022). Even still, like cluster-based randomization in the network setting, switchback experiments suffer from decreased power via a reduced effective sample size. Recent work by Ni, Bojinov, and Zhao (2023) explores the use of spacial clustering and temporal balance to overcome this problem.

In marketplaces based on auctions (e.g., eBay auctions (Blake and Coey 2014) or advertising auctions (Liu, Mao, and Kang 2021)), interference due to shared resources is also a problem. A treatment that encourages higher bidding will lead to treated users winning more auctions and control users necessarily losing them. This leads to a “cannibalization bias” whereby the margin by which the treatment looks better than the control is exaggerated, because when the treatment wins, the control must lose. Switchback experiments have been proposed as a means to mitigate such bias, but their limitations (described above) have led to the development of more tailored experimental designs for online auctions. For example, Liu, Mao, and Kang (2021) propose budget-split designs that eliminate the opportunity for cannibalization bias by splitting the available resources (i.e., the budget) equally and independently between the treatment and control groups. With the resources no longer shared, there exists no imbalanced competition for them.

The works described above seek to eliminate (or at least minimize) interference through the experiment’s design so that the traditional difference-in-means estimator yields unbiased estimates of the ATE. However, an alternative paradigm exists in which interest lies in de-biasing through the analysis of the experiment by modeling the interference rather than eliminating it. For instance, Bui, Steiner, and Stevens (2023) develop a class of general additive network effect models that facilitate unbiased ATE estimation while flexibly modeling network influence. Many other such network modeling approaches exist. See, for example, Parker, Gilmour, and Schormans (2017), Basse and Airoldi (2018), Pokhilko et al. (2019), Koutra, Gilmour, and Parker (2021), and Zhang and Kang (2022) and Section SM3 of the supplementary material for a deeper discussion of such methods. Likewise, Johari et al. (2022b), and Li et al. (2022) develop stochastic market models to capture interference dynamics in two-sided marketplaces. Such methods, if the interference is accurately modeled, enjoy the benefit of increased power by permitting randomization at the user-level. However, accurately modeling interference is nontrivial.

7 Conclusion

The value of experimentation and the accompanying philosophy of trial and error has been observed in many facets of society (Manzi 2012), and its positive impacts in the realm of business in particular are remarkable (Koning, Hasan, and Chatterji 2022). Online controlled experiments are vital tools utilized hundreds of times a day by companies whose products touch the lives of billions (Kohavi et al. 2013; Xu et al. 2015; Google 2022). As many vital societal functions shift online at an unprecedented rate, online experimentation has already found applications outside the mainstream spheres of technology and e-commerce. OCEs have been used to optimize political advertisements and increase user engagement with campaign platforms during the Obama and Trump elections (Christian 2012; Bump 2019). Decision-making tools streamlined by OCEs help clinicians make safer, more cost effective decisions regarding patient care (Austrian et al. 2021). OCEs have also been deployed to identify the psychological impacts of social media on younger demographics (Isaac 2021). Along with the significant growth and popularity of careers under the evolving “data science” profession, online experimentation is almost certainly going to become a common tool for online businesses of all sizes (Schroeder 2021). Given the breadth and depth of OCE applications, we believe that solving the research challenges presented in this review will improve the quality of data-driven decision making in online businesses across the applied domain.

We conclude this literature review with a call to action for greater collaboration between industry and academic statisticians to address the research challenges presented by online experimentation. While this article may be one of the first to provide a cohesive review of the OCE statistics literature, the need for increased cooperation between industry and academia already has been explicitly stated by experts at 13 leading organizations that run online experiments (Gupta et al. 2019). Collaborative partnerships between academia and industry do exist in this space (see, e.g., Waudby-Smith et al. 2021; Ham et al. 2022, respectively for partnerships between Carnegie Mellon and Adobe and Harvard and Netflix on the problem of sequential experimentation). However, the academic statistics community in general seems to lack familiarity with—and access to—this research area. The purpose of this review, therefore, was to introduce academicians to the context and goals of online experimentation, as well as to provide examples and broad, technical discussion of the statistical methodologies regarding sensitivity, effect size, heterogeneity, long-term effects, optional stopping, and interference. In the absence of direct collaboration with industry, it is difficult to develop and test novel methodology without access to data. While some open access repositories exist (see, e.g., Liu et al. 2021; Matias et al. 2021), the proprietary nature of these experiments makes open-access data-sharing uncommon. This is admittedly a challenge and a remaining open problem for research in this space.

Supplementary Materials

The supplementary materials file contains additional discussion of some topics not considered in the paper (e.g., ethics, observational causal inference, and Bayesian methods) as well as expanded discussion of certain topics from the paper (e.g., the relationship between stratified sampling and CUPED, surrogate methods for long-term treatment effect estimation, and network A/B testing).

Acknowledgments

The authors thank Art Owen, Georgi Georgiev, and Somit Gupta for helpful comments on an earlier draft of this article.

Disclosure Statement

The authors report there are no competing interests to declare.