Primary Submission Category: Design of Experiments
AI-assisted design and analysis of experiments with unstructured treatments
Authors: Eli Ben-Michael, Zach Branson,
Presenting Author: Eli Ben-Michael*
Randomized experiments with unstructured treatments—such as text or images—are common in social science research. However, isolating the causal effect of a focal attribute (e.g., the style of text or facial features in images) is challenging because the attribute is typically correlated with other, non-focal attributes of the treatments. While AI technology could be used in an attempt to change focal attributes of a treatment while keeping all non-focal attributes identical, it offers no guarantees that non-focal attributes are not inadvertently changed in the process, such that confounding can still be an issue. We develop a framework for designing and analyzing experiments that target the isolated effect of a binary attribute of unstructured treatments. We consider designs where treatments are drawn from arbitrary distributions—including hand-crafted treatments, existing databases, or AI systems—and we map the bias of the difference-in-means estimator to the discrepancy in non-focal attributes across treatment arms. We develop a procedure that minimizes this bias via a second-stage rejection sampler that adjusts for observable imbalances in non-focal attributes, without assuming the original distributions correctly isolate the focal attribute. For analysis, we show how to conduct asymptotic inference for the difference-in-means estimator in a finite population setting, where inference is justified by the randomization of treatment. We also develop a calibrated model