Primary Submission Category: Heterogeneous Treatment Effects
Designing Replication Studies to Evaluate Sources of Effect Heterogeneity
Authors: Vivian Wong, Peter Steiner,
Presenting Author: Vivian Wong*
Despite interest by funding agencies to promote replication studies for identifying sources of effect heterogeneity, there is not yet consensus on what replication is and how these studies should be conducted. This paper addresses these challenges by providing a formal understanding of causal replication through the potential outcomes framework. We describe 5 assumptions for two or more studies to identify the same causal estimand. The assumptions may be understood as replication design requirements and individual study design requirements to identify a causal effect. Replication failure occurs when one or more replication and/or individual design assumptions are not met. An advantage of this approach is that it is straight-forward to derive research designs for replication. Research designs for direct replication examine whether studies with the same well-defined causal estimand yield the same effect. Research designs for conceptual replications examine whether studies with potentially different causal estimands yield the same effect. Within-study comparisons, multi-site, multi-arm treatment, stepped-wedge, and switching replication designs are common research designs that have not yet been recognized as replication designs. The paper demonstrates that high-quality replication designs for identifying sources of effect heterogeneity are feasible, ethical, and desirable in field settings.