Primary Submission Category: Bayesian Causal Inference
A Bayesian approach to risk constrained iterative experiments
Authors: Yufan Li, Jialiang Mao, Iavor Bojinov,
Presenting Author: Yufan Li*
Our work provides a theoretical foundation for phased releases, a widely adopted practice in the technology sector whereby a firm gradually releases a new product or update through a sequence of A/B tests. Typically, when performing a phased release, the analyst starts by releasing the new updates to a small percentage of the users (i.e., the treatment group); if the treatment is deemed not to cause harm, more users are added to the treatment group. This process continues until either the treatment is estimated as superior to the control, in which case the treatment is deployed to all users, or not, in which case all users are returned to the control. A key design question is how to determine the treatment group size that balances the risk associated with releasing unpopular products with the need to iterate quickly. To solve this problem, we propose a Bayesian approach to determine the treatment group size at each stage under a user-set risk budget that adopts to the observed data. Our method quantifies the risk in terms of the treatment effect of treated users under the Neyman-Rubin potential outcome framework. The treatment group size is determined by controlling the probability of exceeding the risk budget (risk level) below a user-set threshold (risk tolerance). Our approach involves decomposing risk tolerance to each stage by a recursive relation and deriving an upper bound of the stage-wise risk level so that the treatment group size can be analytically solved using a