Primary Submission Category: Mediation
Unpack the mechanisms of treatment effects in AB tests
Authors: Can Cui,
Presenting Author: Can Cui*
Treatment effects measured in AB tests provide an objective and valuable way to evaluate performance of product changes or engineering improvements, and to guide launch decisions. While standard estimates of treatment effect is important, we sometimes run in to the scenario where treatment effects are counter intuitive, or difficult to explain. In the space of streaming experiences, we run thousands of tests to validate and evaluate various engineering changes. Better understanding of mechanisms of measured treatment effects can help tremendously to further improve engineering interventions and success rate of future tests. As most tests involve complex changes (e.g. involving changes in multiple parts of the system) and lead to changes in customer experience in multiple dimensions, we leverage various methods to unpack the underlying mechanisms of the treatment effects, including instrumental variables method, decomposition analysis, and mediation analysis. We will discuss the pro and con of these approaches, especially highlighting the identification assumptions in different types of AB tests, and showcase real life examples of learning that help engineers tweak future interventions and come up with newer and better tests.