Primary Submission Category: Applications in Physical Sciences, Engineering, Environment and Miscellaneous Applications
Productivity and AI Tools: a causal analysis
Authors: Sarah Brodbeck, Rinaldo Oliveira Junior, Ciro Akiyoshi Higashi,
Presenting Author: Leandro Zanon Siqueira*
In real life, in many situations, when a company decides to start using a solution or tool, randomized controlled tests (RCT) cannot be done. That happens for many reasons, such as the cost of opportunity of not enabling all employees to use the tool or even a strong strategic decision of the company. In this scenario, causal inference techniques allow us to isolate the effect of other confounder variables so that we can estimate the real gain of using the tool, even without a RCT.
In this present study, we use Directed Acyclic Graphs (DAG) to estimate how much faster teams in the largest retail bank of Latin America, Itau Unibanco, can develop software by using GitHub Copilot.
Using DAGs and PyWhy library, we compared teams that used GitHub Copilot with those who didn’t (controlling by confounders and mediator variables) and it was possible to estimate that teams that used the AI tool spent about 11% less time with software development.
With this study, it was possible to understand how much it was worth the investment made in GitHub Copilot licenses by Itau Unibanco. Also, this study created a comparable framework for competitive AI tools, allowing the company to choose solutions that bring more efficiency and that present a better cost-benefit ratio.