Primary Submission Category: Multilevel Causal Inference
Aggregate regression for policy evaluation: cheaper/faster but no less accurate/precise
Authors: Mariel Finucane, Dan Thal,
Presenting Author: Dan Thal*
As the organizers of the 2022 ACIC data challenge, we generated thousands of real-world-like data sets and baked in true causal impacts unknown to participants. Participating teams then competed, using their cutting-edge methods to estimate those effects. In total, 20 teams submitted results from 58 estimators that used a range of approaches. We found several important factors driving performance that are not commonly used in business-as-usual applied policy evaluations. Among these, the most surprising to us and our policy-maker colleagues was that there was no apparent benefit to analyzing large patient-level data sets (N = 300,000 patients) instead of data sets that had been aggregated to the level at which treatment status varied (the primary care practice level, N = 500 practices). Specifically, we found that performance did not vary by patient- versus practice-level analysis among the 58 submitted estimators. And within matched pairs of benchmark estimators that we ran at both the patient and practice levels, we found higher bias, larger RMSE, and wider uncertainty intervals using the disaggregated patient-level data sets, though coverage was slightly closer to nominal. In this talk, we will present follow-up work on this intriguing finding, and share intuition for why bias and power do not suffer due to aggregation. Aggregate regressions can streamline the timelines and budgets of policy evaluations, ultimately making high-quality causal evidence more widely available.