Primary Submission Category: Randomization-based inference
Counternull sets for randomized experiments
Authors: Marie-Abele Bind, Donald Rubin,
Presenting Author: Marie-Abele Bind*
In 1990, the clinical psychologist Paul E. Meehl used the term ”counternull” to describe an unspecified nonnull alternative statistical hypothesis. Four years later, Rosenthal and Rubin introduced the less vague ”counternull value of an effect size” (1994) as ”the nonnull magnitude of effect size that is supported by exactly the same amount of evidence as is the null value of the effect size”. Continuing, evidence was explicitly illustrated using the ratio between the likelihoods of the test statistic at the maximum likelihood estimate of the estimand and at the value of that statistic at the null value of the estimand. This explication is well-defined for parametric models and in situations where asymptotic normality is used as the basis for inference, both of which are common in practice. However, in the context of randomized experiments, an arguably more natural definition of evidence is available, and this definition has the advantage of being both model-free and well-defined sub-asymptotically: a counternull value is a nonnull value that yields the same randomization-based p-value as does the null value. From this perspective, the counternull is rarely a unique scalar but rather a set of values. We illustrate this use and its potential attendant new insights using first a social science randomized experiment with 2,216 units, and then an epigenetic randomized experiment with 17 units.