Accenture
Accenture rolls out GitHub Copilot and studies its effect on developers
Accenture and GitHub ran a study of GitHub Copilot with Accenture developers, combining a randomized trial, telemetry and a survey. GitHub reports higher pull request volume, merge rate and successful builds, but publishes no sample size or duration.
Key figures
Copied exactly from the sources. Copy any figure with its citation.
8.69%
Increase in pull requests1
15%
Increase in pull request merge rate1
84%
Increase in successful builds1
About 30%
Share of Copilot suggestions accepted1
What it took
Cost, team size and timeline were not disclosed in the sources.
The problem
Enterprises that buy coding assistants want evidence of effect on real engineering teams, not only on lab tasks. Accenture and GitHub set out to measure it at scale.
What they built
A study of GitHub Copilot used by Accenture developers across engineering, design and testing roles, from entry level to team management.
How they did it
The study compared two randomly chosen groups of developers: one got Copilot, the other did not. It added three more data sets: delivery measures such as pull requests, merges and successful builds; adoption data across the company, such as how many developers installed it and how often they accepted its suggestions; and a survey of the developers using it.
Results
GitHub reports an 8.69% increase in pull requests, a 15% increase in merge rate and an 84% increase in successful builds. Developers accepted about 30% of suggestions and kept 88% of Copilot-generated characters. In the survey, 90% said they felt more fulfilled with their job. These numbers come from GitHub's write-up, with the limits noted above.
All reported figures
| Measure | Figure | Citation |
|---|---|---|
| Increase in pull requests | 8.69%1 | |
| Increase in pull request merge rate | 15%1 | |
| Increase in successful builds | 84%1 | |
| Share of Copilot suggestions accepted | About 30%1 | |
| Copilot-generated characters retained in the editor | 88%1 | |
| Developers who installed the coding-tool extension the same day they got a license | 81.4%1 | |
| Developers who said they felt more fulfilled with their job (survey) | 90%1 |
Lessons for your project
Measure adoption as well as output. The study tracked same-day installs, first accepted suggestion and acceptance rate alongside pull requests and builds.
Combine telemetry with a survey, and label which is which. Satisfaction numbers are self-reported and should not be mixed with delivery numbers.
Ask for sample size and duration before copying a vendor's headline numbers. This write-up does not include them.
Caveats
- Published by GitHub, which sells Copilot, with GitHub and Microsoft staff among the authors. It is vendor-reported.
- The write-up gives no sample size, study duration or statistical significance, and does not separate the randomized results from adoption and survey data.
- The date of the Accenture rollout is not stated. deployedDate uses the publication month as the earliest verifiable date.
Sources
- Research: Quantifying GitHub Copilot's impact in the enterprise with Accenture
GitHub Blog · 2024-05-13 · Company blog · github.blog
Summarized in our own words from the sources above. Spotted an error? See corrections.