Primary Submission Category: Sensitivity Analysis
Quantifying the Robustness of Inferences to Replacement of Data: Applications to Main Effects and Moderators
Authors: Kenneth Frank, Ran Xu, Qinyun Lin, Spiro Maroulis,
Presenting Author: Kenneth Frank*
One of the most important factors affecting the use of evidence for policy or practice is uncertainty of study results. Conventional and new methods attempt to mitigate against sources of uncertainty due to sampling error, systematic bias, or heterogeneous treatment effects. Here we acknowledge that while there have been improvements, there will always be uncertainty regarding an inference. The issue then is to discuss inferences in clear precise terms. Therefore, we characterize uncertainty by quantifying how much the data would have to change to nullify an inference. Specifically, we present the Robustness of Inference to Replacement (RIR) and extend it to interaction or moderating effects. To nullify the inference of an effect of the Early Vocabulary Tier 2 Intervention (EVI) from Coyne et al.’s randomized experiment, one would expect to have to replace 88% (roughly 1264) cases with children for whom the intervention had no effect. To nullify the inference of an interaction of the intervention with baseline vocabulary, a special concern for Tier 2 literacy interventions, one would have to replace 37% (about 528) cases with cases for which the effect of the intervention did not depend on baseline vocabulary. We interpret these sensitivity analyses relative to the strengths and weaknesses of study design for estimating main and interaction effects.