Novelty and Primacy: A Long-Term Estimator for Online Experiments

References

• Abadie, A. (2005), “Semiparametric Difference-in-Differences Estimators,” The Review of Economic Studies, 72, 1–19. DOI: 10.1111/0034-6527.00321.
   Web of Science ®Google Scholar
• Anderson, N. H., and Barrios, A. A. (1961), “Primacy Effects in Personality Impression Formation,” The Journal of Abnormal and Social Psychology, 63, 346–350. DOI: 10.1037/h0046719.
   PubMed Web of Science ®Google Scholar
• Athey, S., and Imbens, G. W. (2006), “Identification and Inference in Nonlinear Difference-in-Differences Models,” Econometrica, 74, 431–497. DOI: 10.1111/j.1468-0262.2006.00668.x.
   Web of Science ®Google Scholar
• Bartolomeo, P. (1997), “The Novelty Effect in Recovered Hemineglect,” Cortex, 33, 323–333. DOI: 10.1016/S0010-9452(08)70008-X.
   PubMed Web of Science ®Google Scholar
• Belton, C. A., and Sugden, R. (2018), “Attention and Novelty: An Experimental Investigation of Order Effects in Multiple Valuation Tasks,” Journal of Economic Psychology, 67, 103–115. DOI: 10.1016/j.joep.2018.05.002.
   Web of Science ®Google Scholar
• Bush, R. R., and Mosteller, F. (1951), “A Mathematical Model for Simple Learning,” Psychological Review, 58, 313–323. DOI: 10.1037/h0054388.
   PubMed Web of Science ®Google Scholar
• Chen, N., Liu, M., and Xu, Y. (2019), “How a/b Tests could Go Wrong: Automatic Diagnosis of Invalid Online Experiments,” in Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pp. 501–509.
   Google Scholar
• Conley, T. G., and Taber, C. R. (2011), “Inference with Difference-in-Differences with a Small Number of Policy Changes,” The Review of Economics and Statistics, 93, 113–125. DOI: 10.1162/REST_a_00049.
   Web of Science ®Google Scholar
• Czitrom, V. (1999), “One-Factor-at-a-Time versus Designed Experiments,” The American Statistician, 53, 126–131.
   Web of Science ®Google Scholar
• Dimick, J. B., and Ryan, A. M. (2014), “Methods for Evaluating Changes in Health Care Policy: The Difference-in-Differences Approach,” JAMA, 312, 2401–2402. DOI: 10.1001/jama.2014.16153.
   PubMed Web of Science ®Google Scholar
• Dmitriev, P., Frasca, B., Gupta, S., Kohavi, R., and Vaz, G. (2016), “Pitfalls of Long-Term Online Controlled Experiments,” in 2016 IEEE International Conference on Big Data, pp. 1367–1376. IEEE.
   Google Scholar
• Dmitriev, P., Gupta, S., Kim, D. W., and Vaz, G. (2017), “A Dirty Dozen: Twelve Common Cetric Interpretation Pitfalls in Online Controlled Experiments,” in Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1427–1436.
   Google Scholar
• Donald, S. G., and Lang, K. (2007), “Inference with Difference-in-Differences and other Panel Data,” The Review of Economics and Statistics, 89, 221–233. DOI: 10.1162/rest.89.2.221.
   Web of Science ®Google Scholar
• Estes, W. K. (1950), “Toward a Statistical Theory of Learning,” Psychological Review, 57, 94–107. DOI: 10.1037/h0058559.
   Web of Science ®Google Scholar
• Feddersen, A., Maennig, W., Borcherding, M. (2006), “The Novelty Effect of New Soccer Stadia: The Case of Germany,” International Journal of Sport Finance, 1, 174–188.
   Web of Science ®Google Scholar
• Gupta, S., Kohavi, R., Deng, A., Omhover, J., and Janowski, P. (2019), “A/b Testing at Scale: Accelerating Software Innovation,” in Companion Proceedings of The 2019 World Wide Web Conference, pp. 1299–1300. DOI: 10.1145/3308560.3320093.
   Google Scholar
• Haizler, T., and Steinberg, D. M. (2020), “Factorial Designs for Online Experiments,” Technometrics, 63, 1–12. DOI: 10.1080/00401706.2019.1701556.
   Web of Science ®Google Scholar
• Hohnhold, H., O’Brien, D., and Tang, D. (2015), “Focusing on the Long-Term: It’s Good for Users and Business,” in Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1849–1858.
   Google Scholar
• Howard, D. R., and Crompton, J. L. (2003), “An Empirical Review of the Stadium Novelty Effect,” Sport Marketing Quarterly, 12, 111–116.
   Google Scholar
• Imbens, G. W., and Rubin, D. B. (2015), Causal Inference in Statistics, Social, and Biomedical Sciences, Cambridge: Cambridge University Press.
   Google Scholar
• Jones, E. E., Rock, L., Shaver, K. G., Goethals, G. R., and Ward, L. M. (1968), “Pattern of Performance and Ability Attribution: An Unexpected Primacy Effect,” Journal of Personality and Social Psychology, 10, 317–340. DOI: 10.1037/h0026818.
   Web of Science ®Google Scholar
• Kaufman, R. L., Pitchforth, J., and Vermeer, L. (2017), “Democratizing Online Controlled Experiments at booking.com,” arXiv preprint arXiv:1710.08217.
   Google Scholar
• Kohavi, R., Deng, A., Frasca, B., Longbotham, R., Walker, T., and Xu, Y. (2012), “Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained,” in Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 786–794. DOI: 10.1145/2339530.2339653.
   Google Scholar
• Kohavi, R., and Longbotham, R. (2017), “Online Controlled Experiments and a/b Testing,” Encyclopedia of Machine Learning and Data Mining, 7, 922–929.
   Google Scholar
• Kohavi, R., Longbotham, R., Sommerfield, D., and Henne, R. M. (2009), “Controlled Experiments on the Web: Survey and Practical Guide,” Data Mining and Knowledge Discovery, 18, 140–181. DOI: 10.1007/s10618-008-0114-1.
   Web of Science ®Google Scholar
• Kohavi, R., and Round, M. (2004), “Emetrics Summit 2004: Front Line Internet Analytics at amazon.com,” Amazon.com, Copyright.
   Google Scholar
• Kohavi, R., Tang, D., and Xu, Y. (2020), Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge: Cambridge University Press.
   Google Scholar
• Li, C. (2010), “Primacy Effect or Recency Effect? A Long-Term Memory Test of Super Bowl Commercials,” Journal of Consumer Behaviour: An International Research Review, 9, 32–44.
   Web of Science ®Google Scholar
• Li, P. L., Dmitriev, P., Hu, H. M., Chai, X., Dimov, Z., Paddock, B., Li, Y., Kirshenbaum, A., Niculescu, I., and Thoresen, T. (2019), “Experimentation in the Operating System: The Windows Experimentation Platform,” in 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 21–30. IEEE. DOI: 10.1109/ICSE-SEIP.2019.00011.
   Google Scholar
• Mutsuddi, A. U., and Connelly, K. (2012), “Text Messages for Encouraging Physical Activity: Are they Effective after the Novelty Effect Wears Off?” in 2012 6th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth) and Workshops, pp. 33–40. IEEE.
   Google Scholar
• Peterson, C. R., and DuCharme, W. M. (1967), “A Primacy Effect in Subjective Probability Revision,” Journal of Experimental Psychology, 73, 61–65. DOI: 10.1037/h0024139.
   PubMedGoogle Scholar
• Poppenk, J., Köhler, S., and Moscovitch, M. (2010), “Revisiting the Novelty Effect: When Familiarity, not Novelty, Enhances Memory,” Journal of Experimental Psychology: Learning, Memory, and Cognition, 36, 1321–1330.
   PubMed Web of Science ®Google Scholar
• Roth, J., and Sant’Anna, P. H. (2021), “Efficient Estimation for Staggered Rollout Designs,” arXiv preprint arXiv:2102.01291.
   Google Scholar
• Sadeghi, S., Chien, P., and Arora, N. (2020), “Sliced Designs for Multi-Platform Online Experiments,” Technometrics, 62, 387–402. DOI: 10.1080/00401706.2019.1647288.
   Web of Science ®Google Scholar
• Scott, S. L. (2010), “A Modern Bayesian Look at the Multi-Armed Bandit,” Applied Stochastic Models in Business and Industry, 26, 639–658. DOI: 10.1002/asmb.874.
   Web of Science ®Google Scholar
• Scott, S. L. (2015), “Multi-Armed Bandit Experiments in the Online Service Economy,” Applied Stochastic Models in Business and Industry, 31, 37–45. DOI: 10.1002/asmb.2104.
   Web of Science ®Google Scholar
• Shen, M., and Lee, M. (2017), “Selection Bias in Online Experimentation,” Available at https://medium.com/airbnb-engineering/selection-bias-in-online-experimentation-c3d67795cceb.
   Google Scholar
• Tan, L., and Ward, G. (2000), “A Recency-based Account of the Primacy Effect in Free Recall,” Journal of Experimental Psychology: Learning, Memory, and Cognition, 26, 1589–1625.
   PubMed Web of Science ®Google Scholar
• Tang, D., Agarwal, A., O’Brien, D., and Meyer, M. (2010), “Overlapping Experiment Infrastructure: More, Better, Faster Experimentation,” in Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 17–26.
   Google Scholar
• Thorndike, E. L. (1898), “Animal Intelligence: An Experimental Study of the Associative Processes in Animals,” The Psychological Review: Monograph Supplements, 2, i–109. DOI: 10.1037/h0092987.
   Google Scholar
• Urban, S., Sreenivasan, R., and Kannan, V. (2016), “It is all A/Bout Testing: The Netflix Experimentation Platform,” https://medium.com/netflix-techblog/its-all-a-bout-testing-the-netflixexperimentation-platform-4e1ca458c15.
   Google Scholar
• Van Erkel, P. F., and Thijssen, P. (2016), “The First One Wins: Distilling the Primacy Effect,” Electoral Studies, 44, 245–254. DOI: 10.1016/j.electstud.2016.09.002.
   Web of Science ®Google Scholar
• Wang, Y., Gupta, S., Lu, J., Mahmoudzadeh, A., and Liu, S. (2019), “On Heavy-User Bias in a/b Testing,” in Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 2425–2428. DOI: 10.1145/3357384.3358143.
   Google Scholar
• Wu, C. J., and Hamada, M. S. (2011), Experiments: Planning, Analysis, and Optimization (Vol. 552), New York: Wiley.
   Google Scholar