Primary Submission Category: Causal Inference and Bias/Discrimination
Machine Learning Regression Adjustment for RCTs with Survey Outcomes
Authors: Alex Whitworth,
Presenting Author: Alex Whitworth*
Randomized Controlled Trials (RCTs) with survey outcomes are common in industry. These studies present two challenges to the practitioner: (i) the survey completion population may be unbalanced compared to the treated population; and (ii) there may be systematic biases in survey non-completion. In this work, we implement and evaluate a generalized Oaxaca-Blinder estimator (Guo, Basse; 2023) using post-stratification weighting. We evaluate this estimator via an extension of the dowhy framework (Sharma, Kiciman; 2020) where we simulate binary outcomes with binary survey completion. The Oaxaca-Blinder estimator is compared to the standard difference in means estimator. We find that the Oaxaca-Blinder estimator produces unbiased estimates with ~1/3rd narrower confidence intervals and is robust to (i) correlation of survey completion with survey outcome; (ii) unobserved confounders; and (iii) data subset validation. However our simulations show some problems with placebo tests. This work is consistent with prior work showing that variance reduction in RCTs is achievable using machine learning regression adjustment estimators and that the variance reduction is achievable for RCTs with survey outcomes.