JetBrains
JetBrains' in-house Mellum model lifts code written by completion from 0.25 to 0.39 in Python
JetBrains trained its own 4-billion-parameter model, Mellum, to power cloud code completion in its developer tools. JetBrains authors report that in production the share of code written with completion rose in every language measured, for example from 0.25 to 0.39 in Python, against the smaller local model.
Key figures
Copied exactly from the sources. Copy any figure with its citation.
0.39
(cloud) versus 0.25 (local)
Ratio of Completed Code (share of code written with completion) in production, Python: cloud completion versus JetBrains' smaller local model1
0.49
(cloud) versus 0.42 (local)
Ratio of Completed Code in production, Java: cloud completion versus local model1
0.45
(cloud) versus 0.31 (local)
Ratio of Completed Code in production, PHP: cloud completion versus local model1
0.42
(cloud) versus 0.28 (local)
Ratio of Completed Code in production, JS/TS: cloud completion versus local model1
What it took
Cost, team size and timeline were not disclosed in the sources.
The problem
Developers expect their coding tools to finish lines and blocks of code as they type. Most tools of this kind rely on closed services from other vendors or on large chat models, which JetBrains says are costly and slow for typing-speed use and have licences that complicate redistribution. JetBrains also already shipped a small completion model that runs on each user's computer, and wanted higher quality without the delay of a large model.
What they built
JetBrains trained Mellum, a 4-billion-parameter model built only for code completion, and uses it for cloud completion in its developer tools. It was trained from scratch on code with permissive licences. The released version is open-weight under the Apache-2.0 licence (the files are published, so anyone can run and adapt them). The paper says the models running in production are similar to the released ones but not identical.
How they did it
Training had several stages: broad code pre-training, then training on filling in the middle of a file, then training with project context drawn from other files, then a preference stage that discourages unhelpful output such as empty statements and endless generation. The team kept the model small so that it responds fast and fits on cost-efficient GPUs. Quality was checked offline on benchmarks and online from user telemetry in the tools.
Results
In production telemetry, the share of code written with completion was higher for the cloud model than for the local model in all nine languages, for example 0.39 against 0.25 in Python and 0.49 against 0.42 in Java. The share of suggestions accepted was higher in most languages, such as Python at 0.34 against 0.26, but equal in Java and Rust and lower in Go and PHP. A single request takes 400 ms on the backend, 700 ms at peak.
What to copy
If your use case is narrow and latency matters, test a small purpose-built model before a large general one. Measure it on live usage and keep both a usage metric and a quality metric, because they can disagree.
All reported figures
| Measure | Figure | Citation |
|---|---|---|
| Ratio of Completed Code (share of code written with completion) in production, Python: cloud completion versus JetBrains' smaller local model | 0.39 (cloud) versus 0.25 (local)1 | |
| Ratio of Completed Code in production, Java: cloud completion versus local model | 0.49 (cloud) versus 0.42 (local)1 | |
| Ratio of Completed Code in production, PHP: cloud completion versus local model | 0.45 (cloud) versus 0.31 (local)1 | |
| Ratio of Completed Code in production, JS/TS: cloud completion versus local model | 0.42 (cloud) versus 0.28 (local)1 | |
| Acceptance rate (share of shown suggestions that users accepted) in production, Python: cloud versus local model | 0.34 (cloud) versus 0.26 (local)1 | |
| Acceptance rate in production, Java: cloud versus local model | 0.34 (cloud) versus 0.34 (local)1 | |
| Time spent on a single completion request on the backend, typical and at peak working-day times | 400 ms (typical), 700 ms (peak)1 |
Lessons for your project
A small model built for one job can be enough. JetBrains chose a 4-billion-parameter model so that it responds fast and fits on cost-efficient GPUs, and says a bigger architecture might score higher offline but would be slower.
Judge quality on real use, not only on test sets. JetBrains tracks how much code users actually write with completion and how many suggestions they accept.
Check what a metric hides. Acceptance rate was flat for Java and lower for Go and PHP in the cloud model, even though the share of completed code rose, so read both.
Weigh licensing and governance when you pick a model. JetBrains says restrictive licences and unclear training data on available models pushed it to train its own on permissively licensed code.
Caveats
- All figures come from a preprint by JetBrains employees, about JetBrains' own product. No outside party has verified them.
- The production models are similar to, not identical to, the released Mellum models. The paper says production models were trained on about 3 trillion tokens against 4 trillion for the open-sourced ones, and that offline results are almost the same. Only the production telemetry is used here.
- The comparison is between the cloud model and a different, local completion model about 40 times smaller. It is not a comparison against no AI, and the paper does not describe the user groups or period, so differences may not come from the model alone.
- Ratio of Completed Code counts characters written with completion, including the IDE's built-in completion, as a share of all code written. It measures how much code completion produces, not developer time saved or code quality.
- Acceptance rate in the cloud model is lower than local for Go (0.38 versus 0.42) and PHP (0.31 versus 0.32), and equal for Java and Rust. The paper says cloud completion typically raises it.
- Neither source gives a go-live date. The deployed date is shown as 2025 only because the paper, posted in October 2025, reports the cloud model already in production; it may have gone live earlier. A JetBrains blog post says Mellum was trained from scratch to power cloud completion, but the page prints no date.
- The paper says the cloud service has hundreds of thousands of users in one place and millions in another. Volumes are not an outcome.
Sources
- Mellum: Production-Grade in-IDE Contextual Code Completion with Multi-File Project Understanding
arXiv (JetBrains authors) · 2025-10-07 · Research paper · arxiv.org
- Mellum Goes Open Source: A Purpose-Built LLM for Developers, Now on Hugging Face
The JetBrains Blog · Undated · Company blog · blog.jetbrains.com
Summarized in our own words from the sources above. Spotted an error? See corrections.