Primary Submission Category: Causal Inference and SUTVA/Consistencies Violations
Operational Challenges in Scaling Randomized Trials: The Role of Capacity Constraints
Authors: Hannah Li, Justin Boutilier, Jonas Jonasson,
Presenting Author: Hannah Li*
A concern about service interventions, commonly found in domains like public health and education, is that promising interventions at the randomized controlled trial (RCT) stage may not perform well at scale. Although many factors contribute to this difficulty in scaling, in this work we highlight and isolate the effects of an operational factor: capacity constraints.
If an intervention requires a service that is capacity constrained, then participants in the RCT may face a waiting time that depends on the number of providers and number of other participants in the system. We show that this dependency may lead to violations of the Stable Unit Treatment Value Assumption (SUTVA) and creates scaling issues.
We consider a case study of a mobile health platform designed to improve patients’ adherence to tuberculosis treatment. By modeling patients’ interactions as a queueing system, we demonstrate that the effects observed in an RCT may decrease when scaling up to a larger patient population, due to the system’s limited capacity.
Furthermore, we find a counterintuitive implication for conventional power analysis: increasing the sample size of an RCT without appropriately expanding capacity can paradoxically decrease the study’s power. To address this, we leverage principles from operations research and introduce a method for joint power and capacity analysis that leverages the underlying structure of these interventions in order to increase power.