Sliced Designs for Multi-Platform Online Experiments

Abstract

Multivariate testing is a popular method to improve websites, mobile apps, and email campaigns. A unique aspect of testing in the online space is that it needs to be conducted across multiple platforms such as a desktop and a smartphone. The existing experimental design literature does not offer precise guidance for such a multi-platform context. In this article, we introduce a multi-platform design framework that allows us to measure the effect of the design factors for each platform and the interaction effect of the design factors with platforms. Substantively, the resulting designs are of great importance for testing digital campaigns across platforms. We illustrate this in an empirical email application to maximize engagement for a digital magazine. We introduce a novel “sliced effect hierarchy principle” and develop design criteria to generate factorial designs for multi-platform experiments. To help construct such designs, we prove a theorem that connects the proposed designs to the well-known minimum aberration designs. We find that experimental versions made for one platform should be similar to other platforms. From the standpoint of real-world application, such homogeneous subdesigns are cheaper to implement. To assist practitioners, we provide an algorithm to construct the designs that we propose.

KEYWORDS:

• A/B testing
• Blocking
• Design of experiments
• Digital marketing
• Email testing
• Web experiments

Acknowledgments

The authors thank the Editor, Associate Editor and referees for useful comments and the IMC and Alumni Relations teams at the Wisconsin School of Business for their generous help with the Update Magazine study for this research.

Notes

1 Please refer to Cheng, Steinberg and Sun (1999) for a formal definition of estimation capacity.