Uploaded October 2020 | Updated September 2026, 9 hours ago
A/B testing on the web is a popular way to make data-driven decisions when designing a website. However popular A/B testing may be, it’s easy to design an experiment that results in poor insights and a false sense of objectivity. One way to get poor insights is by changing the proportion of users assigned to arms (aka “treatments”) during an experiment without considering this in the analysis. This is often done on the web with assignment ramp-up or multi-armed bandits. In this talk I will discuss the problem, a solution, and considerations.
A/B testing on the web is a popular way to make data-driven decisions when designing a website. However popular A/B testing may be, it’s easy to design an experiment that results in poor insights and a false sense of objectivity. One way to get poor insights is by changing the proportion of users assigned to arms (aka “treatments”) during an experiment without considering this in the analysis. This is often done on the web with assignment ramp-up or multi-armed bandits. In this talk I will discuss the problem, a solution, and considerations.
