Primary Submission Category: Randomized Designs and Analyses
Design-Aware Variance Reduction for Switchback Experiments: A Comparative Study
Authors: Sergei Pankratev,
Presenting Author: Sergei Pankratev*
Switchback experiments and other clustered randomized designs are widely used on online platforms, but the clustered, time-dependent nature of these designs can make standard variance reduction methods behave differently than in standard A/B tests. We evaluate design-aware variance reduction methods for switchbacks—CUPED, CUPAC (ML-based covariate adjustment), and doubly robust (DR) estimators—relative to a baseline switchback analysis with cluster-robust standard errors. Through a hierarchical simulation framework that varies key regime parameters—number of clusters, cluster-size imbalance, within-cluster autocorrelation, carryover, and predictive signal strength—we evaluate validity (false positive rate and confidence interval coverage) and efficiency (standard error reduction, power, and minimum detectable effect as a function of run length). We also include a sensitivity analysis for cross-cluster spillovers to quantify bias and inference degradation under mild interference. The primary outcome is a practitioner-oriented regime map: when CUPED, CUPAC, or DR are most beneficial, and when time and cluster dependence and finite-cluster effects limit improvements.