Primary Submission Category: Machine Learning and Causal Inference
Adaptive designs for best arm identification and evaluation, with application to vaccine confidence messaging
Authors: Molly Offer-Westort, Leah Rosenzweig,
Presenting Author: Molly Offer-Westort*
This paper contributes to the literature on adaptive experimental designs for best arm identification. In particular, we consider a setting where the experimenter wishes not only to learn the best arm, but also to evaluate response under that arm, or treatment effects with respect to the best arm and a control condition. Adaptive designs can help experimenters more efficiently learn which treatment condition among a set of alternatives will perform best on a given response measure. However, naive frequentist estimates of response under the best arm will be upwardly biased if they are calculated on the same data used to learn which arm is best. To resolve this, we consider a two-stage design where best arm identification is learned in an adaptive first stage, and evaluated alongside a control condition in a second stage. We show improvements in simple regret, and bias and precision of treatment effect estimation as compared to alternative designs where learning and evaluation are objectives.
In our application, we use an adaptive best-arm identification algorithm to optimize informational messaging on vaccines in an online study among Facebook users in Kenya and Nigeria. We demonstrate that optimized personalized messaging improves vaccine confidence and intentions to get vaccinated over a uniform public service announcement condition or a pure control condition.