Primary Submission Category: Machine Learning and Causal Inference
Just Trial Once: Ongoing Causal Validation of Updates to Machine Learning Models
Authors: Jacob M Chen, Michael Oberst,
Presenting Author: Jacob M Chen*
The use of machine learning (ML) models as clinician support tools is increasing in popularity. Evaluating the causal impact of deploying such models on clinical outcomes can be done with a randomized control trial (RCT), such as a cluster-randomized trial. However, ML models are inevitably updated over time, and we often lack evidence for the impact of these updates on the same clinical outcomes. While this impact could be repeatedly validated with ongoing RCTs, in practice, such experiments are expensive, time-consuming, and difficult to run. In this work, we present an alternative solution: using only data from a prior RCT that tested other models, we give conditions under which the causal effect of an updated ML model can be precisely estimated or bounded. Our assumptions incorporate two realistic constraints: ML predictions are often deterministic, and their impact depends on clinician trust in the model. Based on our analysis, we give recommendations for cluster randomized trial designs that maximize their ability to assess future versions of an ML model. Our hope is that following our proposed trial design will save practitioners time and resources while allowing for quicker deployments of updates to ML models.