Primary Submission Category: Randomized Studies
Experimental and Quasi-Experimental Identification of Conditional Average Treatment Effects: A Four-Arm Within-Study Comparison
Authors: Bryan Keller, Vivian Wong, Sangbaek Park, Jingru Zhang, Patrick Sheehan, Peter Steiner,
Presenting Author: Bryan Keller*
In a largest-of-its-kind four-arm within-study comparison (WSC), we asked 2200 participants to request to receive either a mathematics or vocabulary training,
recorded their request, and then randomly assigned them to a session. In addition to mathematics and vocabulary outcomes, we collected over 30 baseline
covariates vetted via a pilot study such that we expect unconfoundedness to approximately hold. This design permits both experimental and quasi-experimental identification of the ATE, ATT, and ATC. For example, the ATT for the mathematics training intervention is experimentally identified by the group mean difference (T – C) for those who asked to be assigned to the mathematics group. A quasi-experiment based on self-selection may be created by comparing those who requested mathematics training and received it with those who requested vocabulary training and received it. A number of methods that condition on observed covariates to reduce bias are used to estimate ATEs with the quasi-experimental data (e.g., main effects ANCOVA, PS analysis, AIPW, BART as S-learner, causal forests, TMLE). The study was powered to test for correspondence between experimental and quasi-experimental estimates. A minimum detectable effect size corresponding with 80% bias reduction (RCT vs QED) is used as the equivalence threshold for correspondence testing. Planned analyses are described in detail in an OSF preregistration. Results will be discussed.