Primary Submission Category: Bayesian Causal Inference
Why Even Bayesians Need to Worry about Multiple Comparisons
Authors: George Perrett, Jennifer Hill, Marc Scott, Christopher Buglino,
Presenting Author: George Perrett*
Researchers asking causal questions are often interested not only in the average treatment effect but also subgroup specific treatment effects that allow for more nuanced understanding of who benefits from an intervention. However, this pursuit can lead to issues with multiple comparisons. While previous research has demonstrated that Bayesian methods with regularizing prior distributions are more conservative than their frequentist counterparts and can lead to better outcomes than either ignoring the issue or using corrections (Bonferroni, FDR), the extent to which Bayesian methods eliminate the problem of multiple comparisons has been overstated. Critically, we demonstrate a setting common in social sciences where Bayesian regularizing priors are not sufficient to control false positive claims. Moreover, we show that this is not only limited to false positive claims but extends to sign errors. We characterize this setting as dominated by “shrinkage to the wrong place” and present a diagnostic to guide researchers to more appropriate models.