Primary Submission Category: Randomized Studies
The Plausibility of Experimental Findings under Selective Reporting: An Application to Randomized Experiments on Voter Turnout by Proprietary Organizations
Authors: Thomas Leavitt, Donald Green,
Presenting Author: Thomas Leavitt*
In principle, voter turnout research is well suited to drawing valid causal inferences. Most studies of voter turnout in the past decade randomly assign interventions and rely on administrative data to measure outcomes. Moreover, academic researchers who study voter turnout have been early and earnest adopters of pre-registration, which lessens the threat of publication bias. However, an increasing share of experiments in this domain is conducted by consultants
or staff working within organizations that do not register and disclose all of the studies they conduct. Such selective reporting is a source of the replication crisis, which has led prominent researchers to declare that published research findings are often false. This paper takes up the issue of post-study plausibility of experimental findings under selective reporting. We propose a randomization-based, Bayesian procedure that enables the principled incorporation of assumptions about selective reporting into the prior distribution of the average effect. We pair this procedure with a Bayesian sensitivity analysis whereby researchers can assess the robustness of posterior inferences under increasingly severe degrees of selective reporting. Unlike existing methods, our procedure enables judgments about the plausibility of individual experimental findings under selective reporting, although we also show the implications of our method for meta-analyses that aggregate across multiple experiments.