Primary Submission Category: Causal Inference and SUTVA/Consistencies Violations
Symbiosis bias in A/B tests of Recommendation Algorithms
Authors: Jean Pouget-Abadie, Jennifer Brennan, David M. Holtz,
Presenting Author: David M. Holtz*
One assumption underlying the unbiasedness of global treatment effect estimates from randomized experiments is the stable unit treatment value assumption (SUTVA). Experiments that compare the efficacy of different recommendation algorithms often violate SUTVA, because each algorithm is trained on a pool of shared data produced by the different recommendation algorithms being evaluated in the experiment. This shared training data across recommendation algorithms can lead to serious bias in the routine evaluations of such algorithms. We illustrate the presence and magnitude of this bias, which we call “symbiosis bias,” in a real data study on a large tech platform. We further explore, through simulation, cluster randomized and data-diverted solutions to mitigating this bias, and use a stylized analytical model to characterize the relative efficacy of these two solutions at reducing symbiosis bias under different conditions.