Primary Submission Category: Multilevel Causal Inference
Designing Optimal, Data-Driven Educational Policies from Multisite Randomized Trials
Authors: Youmi Suk, Chan Park,
Presenting Author: Youmi Suk*
Optimal treatment regimes (OTRs) have been popular in computer science and personalized medicine to provide data-driven, optimal recommendation rules to individuals. Unfortunately, much of the methodological work on OTRs focuses on single-site settings and there is little work on tailoring existing OTRs to educational settings where students i.e. the study unit, are nested within schools and there are hierarchical dependencies. The goal of this paper is to design OTRs from multisite randomized trials, which are frequently used in education to evaluate educational programs. We study how to modify popular OTR methods, notably Q-learning and weighting methods, in order to enhance their performances in multisite randomized trials. We consider a total of 12 modifications, 6 modifications for Q-learning and 6 modifications for weighting, based on using different multilevel models, moderators, or augmentations. Through simulation studies, we find that all Q-learning modifications enhance the performance in multisite randomized trials, but the modifications with random treatment effects show the most promise in handling cluster-level moderators. Among weighting methods, the modification that adds cluster dummies into moderator variables and augmentation terms performs the best across simulation conditions. We also demonstrate our proposals using data from a multisite randomized trial in Colombia on evaluating conditional cash transfer programs to maximize educational attainment.