Primary Submission Category: Randomized Studies
Design-Based Confidence Sequences for Anytime-valid Causal Inference
Authors: Dae Woong Ham, Iavor Bojinov, Michael Lindon, Martin Tingley,
Presenting Author: Dae Woong Ham*
Many organizations run thousands of randomized experiments, or A/B tests, to statistically quantify and detect the impact of product changes. Analysts take these results to augment decision-making around deployment and investment opportunities, making the time it takes to detect an effect a key priority. Currently, however, the analysis is only performed at the end of the study. This is undesirable because strong effects can be detected before the end of the study, which is especially relevant for risk mitigation when the treatment effect is negative. Alternatively, analysts could perform hypotheses tests more frequently and stop the experiment when the estimated causal effect is statistically significant, i.e., confidence sequences. Our paper provides valid confidence sequences from the design-based perspective, where we condition on the full set of potential outcomes and perform inference on the obtained sample. Our design-based confidence sequence accommodates a wide variety of sequential experiments in an assumption-light manner. In particular, we build confidence sequences for 1) the average treatment effect, 2) the reward mean difference in bandits, 3) the average contemporaneous treatment effect for time series/panel data settings with potential carryover effects. We further provide a variance reduction technique that incorporates modeling assumptions and covariates. We apply our proposed confidence sequences to experiments conducted by Netflix.