Alex Miller, Operations, Information and Decisions, The Wharton School, and Kartik Hosanagar, Operations, Information and Decisions, The Wharton School
Abstract: In this project, we attempt to provide a rigorous, empirical study of e-commerce A/B testing strategies. We perform a meta-analysis on 2,732 A/B tests conducted by 252 ecommerce companies across seven industries over the course of three years. While there is much interest in the field of digital experimentation generally, little is known empirically about the testing strategies of firms in real-world environments and how these strategies are related to business outcomes. Our dataset gives us unique insight into what firms are experimenting with on their websites and which of these strategies are associated with larger experimental effect sizes. We develop a framework for quantifying the effect of two different experimental factors on an intervention’s ultimate effect size: the type (or content) of an experiment and its location within a website’s conversion funnel. After providing a descriptive analysis of A/B testing practices among the firms in our sample, we exploit the metadata in our dataset to classify the experimental interventions using this framework. We find that experiments involving price promotions and those targeted on category or product listing pages are associated with the largest effect sizes, relative to other experiment types in our sample. We then attempt to identify heterogeneity in the effectiveness of different types of interventions at different stages of the conversion funnel. We find evidence that consumers’ response to different types of promotions depends on where those promotions are targeted within a website’s architecture. In particular, we find that promotional interventions on product prices are most effective early in the conversion funnel, whereas shipping-related promotions are most effective late in the conversion funnel (on product and checkout pages). As a unique, large-scale, cross-firm meta-analysis of empirical experimentation practices, this project not only provides practical insight for managers, but also makes a theoretical contribution to the e-commerce literature by documenting and quantifying how multiple dimensions of website design shape online shopping behavior.
Read the working paper here.