Primary Submission Category: Machine Learning and Causal Inference
A Two-Part Machine Learning Approach to Characterizing Network Interference in A/B Testing
Authors: Yuan Yuan, Kristen Altenburger,
Presenting Author: Yuan Yuan*
Randomized control trials, or “A/B tests”, have been crucial for businesses to understand the impact of new product or user experience changes. However, conventional A/B testing methods are limited in the presence of interference – a unit’s response may be affected by other units’ treatments. The current literature on addressing interference in A/B tests has two major limitations: failing to account for latent network structures of interference and relying on human experts to model interference patterns. To overcome these issues, we propose a two-part machine learning approach to automatically characterize interference based on both local network structures and the treatment assignments among users in their network neighborhood. We construct network motifs with treatment assignment information, referred to as causal network motifs, to characterize the network interference status for each unit. We then develop machine learning approaches, based on decision trees and nearest neighbors, to map each causal network motif representation to an “exposure condition.” We demonstrate the validity of our approach through two sets of experiments: a simulated experiment on Watts-Strogatz network and a 1-2 million user experiment on Instagram. Overall, our approach offers a complementary, automated method that can enhance the capabilities of human experts in addressing interference in A/B testing.