Primary Submission Category: Multilevel Causal Inference
Controlling family-wise error while testing tree-structured hypotheses about treatment effects
Authors: Ajinkya H. Kokandakar, Sameer K. Deshpande,
Presenting Author: Ajinkya H. Kokandakar*
We describe two applications that involve testing hypotheses about hierarchically organized treatment effects. In the first application, we wanted to determine the effect of playing organized youth sports, a treatment condition that could be defined at multiple levels of resolution (e.g. playing any sport, a collision sport, or a particular sport). The second application involved testing for significant differences between treatment effects across subgroups discovered in a data-driven manner. In both problems, hypotheses can be arranged in a binary tree such that if a parent hypothesis is false, at most one of its two children hypotheses can be true. We develop an ordered testing procedure that exploits these logical relationships to control family-wise error rate (FWER). Briefly, we keep testing along a path in the tree until we reach the first failed rejection of a null hypothesis. The key innovation in our approach is the allocation of the significance level for individual tests in a way that maintains FWER across any configuration of true hypotheses. Our procedure allows us to adaptively determine the resolutions of treatment for which there are significant effects. Similarly, we can adaptively identify subgroups exhibiting significant treatment effect differences.