Primary Submission Category: Randomized Studies
Design and Analysis of Temporal Experiments in Ride-Hailing Platforms
Authors: Ruoxuan Xiong, Alex Chin, Sean Taylor,
Presenting Author: Ruoxuan Xiong*
We study the design and analysis of temporal experiments, where an intervention is repeatedly applied to the same set of experimental units over time, and units’ longitudinal observations are available for the estimation of treatment effects. The motivating setting is that a ride-hailing platform tests changes to marketplace algorithms, such as pricing and matching, and estimate effects from longitudinal outcomes, such as the rate at which ride requests are completed, at the city level. The design problem involves the planning of partitioning time periods into intervals and assigning the new intervention at the interval level. We propose an autoregressive specification for the outcomes, from which treatment effects can be efficiently estimated. Based on analyzing the statistical properties of this specification, we show that the efficiency of the design depends on three factors: carryover effects from interventions at earlier times, serial correlation in outcomes, and heterogeneity and periodicity of experiment times. We further propose a meta-analysis approach that leverages massive historical data to construct beliefs about the three factors and optimally design experiments. We further conduct a careful simulation study in realistic settings to build intuition and guidance for practitioners to optimally design temporal experiments.