Primary Submission Category: Generalizability/Transportability
Generalizing Text Experiments to Real-World Contexts
Authors: Victoria Lin, Eli Ben-Michael, Louis-Philippe Morency,
Presenting Author: Victoria Lin*
As natural language processing systems are increasingly deployed in real-world settings, it is important to understand how changes in language affect how readers think, feel, and behave. For instance, we may be interested in how varying the tone of a passage of text affects a reader’s response. Randomized text experiments, where texts are randomly assigned to readers, offer a way to estimate such causal effects free from confounding factors. However, the effect of a text attribute depends on its context: the many other attributes of the text (e.g., topic, formality). Therefore, an effect estimated over a randomized—and necessarily artificial—body of text may not be generalizable to real-world settings. To address this issue, we propose a stochastic intervention framework for generalizing experimental text estimates to any new corpus or distribution of texts. We introduce an empirical importance-weighting estimation approach that leverages large language models to generate robust estimates of real-world text distributions. Through several empirical studies, we demonstrate that the effect of the same text attribute on reader response can in fact vary substantially from one text setting to another.