Primary Submission Category: Regression Discontinuity
Quasi-experimental designs for learning health systems
Authors: Amy Cochran, Valerie Odeh-Couvertier, Gabriel Zayas-Caban, Kenneth Nieser, Brian Patterson,
Presenting Author: Amy Cochran*
At the heart of learning health systems are risk prediction models that are validated, continually updated, and used to guide day-to-day clinical care according to patient risk profiles. While the validation and updating of risk prediction algorithms has been a major area of research, less attention has been paid to the evaluation of the consequences of using a risk prediction model in a learning health system. We developed a causal inference method for evaluating the impact of intervening on a patient within a learning system according to risk predictions. Our method builds on a regression discontinuity (RD) design to estimate (local) average treatment effects in settings when the intervention is determined according to whether a patient’s predicted risk exceeds a certain value. Critically, our method allows for the specific interference that arises in a learning health system, whereby prior patients inform the care of future patients. Local average treatment effects are formally defined and identified, and estimators are proposed and analyzed. The method is tested in a simulated learning health system. Our method is a first step towards new standards for how learning health system should independently evaluate, in real-time, the use of risk predictions models in their day-to-day operations.