Primary Submission Category: Goodness-of-fit
An impossibility result for evaluating the goodness-of-fit of causal models
Authors: James Stratton, Nicolaj Thor,
Presenting Author: James Stratton*
The coefficient of determination ($R^2$), and related measures of goodness-of-fit in a regression analysis, measure the share of variation in an outcome that can be predicted by an analyst observing a given set of independent variables. We ask whether there are corresponding measures of the share of variation causally explained by a given set of independent variables, under the assumption that the analyst has the ability to both observe and experiment with those variables. We argue that any valid measure of goodness-of-fit in a causal model should satisfy three axioms: monotonicity (the measure should not decrease as more variables are observed), completeness (the measure should equal 1 when all variables are observed), and limited information (the measure should be strictly less than 1 when the observed variables fail to fully explain variation in the outcome). Our main result is that these three requirements are incompatible. This incompatibility provides theoretical support for the view that causal inference should focus on parameter estimation rather than goodness-of-fit measurement. Motivated by this result, we develop tools for assessing the relative importance of independent variables in causing variation in any given outcome, by considering various weakenings of our axioms.