Primary Submission Category: Sensitivity Analysis
Sensitivity to Attrition for Inferences from an RCT
Authors: Kenneth Frank, Kenneth Frank, Qinyun Lin,
Presenting Author: Benjamin Cher*
Most randomized field experiments experience some attrition. Moreover, the extent of attrition may differ by treatment condition in systematic, non-random ways, biasing estimates of treatment effects and contributing to invalid inferences. We address concerns about non-random attrition by quantifying the conditions necessary in the attritted data to nullify an inference based on observed data. We do so non-parametrically by quantifying what must be the overall mean in the attritted data and the estimated treatment effect in the attritted data to nullify an inference. We also derive results based on a correlational framework, deriving what the correlation between any predictor and outcome must be in the attritted data to nullify an inference. In the empirical example, we show that the attritted students in the Tennessee Class Size study would have to have experienced a negative effect of small classes to nullify the inference that small classes have positive effects on achievement. While this provides no certainty about the inference, it does quantify the conditions necessary to nullify the inference, informing scientific interpretations and corresponding policy debate.