Meta
Meta improves its AI characters in Instagram, WhatsApp and Messenger over 15 Llama 3.1 model versions
Meta's authors describe how they retrained a Llama 3.1 70B chat model 15 times for AI characters that users create and chat with across Instagram, WhatsApp and Messenger. They report that the best new versions lifted engagement in 7-day tests, and that characters broke their instructions far less often.
Key figures
Copied exactly from the sources. Copy any figure with its citation.
up to 8.8%
Best gain in engagement breadth (how many users engage) in a 7-day A/B test of a newly deployed model version, from version 14 in the paper's appendix table1
19.4%
Best gain in engagement depth (how much each user engages) in a 7-day A/B test of a newly deployed model version, from version 13 in the paper's appendix table1
26.6%
to 5.8%
Rate at which characters broke their instructions, judged by an AI reviewer on internal chat tests, from version 2 to version 81
59.2%
to 84.8%
Score on IFEval, a public instruction-following benchmark, from version 2 (before launch) to version 71
What it took
The problem
Meta lets people create and chat with AI characters inside Instagram, Messenger, WhatsApp and the web. Unlike a work assistant, there is no right answer to check. Success means people want to keep chatting, and that is hard to measure and easy to get wrong. Meta wanted a repeatable way to make the chat model more engaging while keeping it on-character and safe.
What they built
Meta's team started from Llama 3.1, its own open-weight model family (open-weight means the model files are published). They built a loop they call CharacterFlywheel. Each round collects real chat data, trains a scoring model to predict what people like, and then retrains the chat model against that score. Trained human raters, not only user clicks, were the main source of the preference signal. Only the 70B size was run in production.
How they did it
Meta made 15 versions between January 2024 and April 2025. The first seven were tested internally before the public launch on July 29, 2024, and the next eight were released to users. Each release was checked against the previous one in a 7-day randomised test with real users. The team also checked safety, instruction following and public benchmarks at each step.
Results
The authors report that 7 of 8 newly deployed models showed positive lift over the baseline. In the 7-day tests, the best result for engagement breadth was up to 8.8% and the best for depth was 19.4%, from two different versions. The share of instruction violations in character chats fell from 26.6% to 5.8%, and the IFEval benchmark score rose from 59.2% to 84.8% before launch. Engagement grew steadily over nine months after launch.
What went wrong
Version 12 optimised too hard against the scoring model, which rated it highly, while real engagement fell, with +0.05% breadth and -2.9% depth. Meta learned that the score should stay below a ceiling and used that rule for later versions.
What to copy
Treat a model for an open-ended product as something you improve in small, tested steps. Keep the control group, set limits on any score the model is trained to chase, and keep a record of failed versions.
All reported figures
| Measure | Figure | Citation |
|---|---|---|
| Best gain in engagement breadth (how many users engage) in a 7-day A/B test of a newly deployed model version, from version 14 in the paper's appendix table | up to 8.8%1 | |
| Best gain in engagement depth (how much each user engages) in a 7-day A/B test of a newly deployed model version, from version 13 in the paper's appendix table | 19.4%1 | |
| Rate at which characters broke their instructions, judged by an AI reviewer on internal chat tests, from version 2 to version 8 | 26.6% to 5.8%1 | |
| Score on IFEval, a public instruction-following benchmark, from version 2 (before launch) to version 7 | 59.2% to 84.8%1 | |
| Engagement lift of version 12, the setback caused by over-optimising against the reward model | +0.05% breadth and -2.9% depth1 |
Lessons for your project
Test every model update on real users against a control group. Meta ran 7-day randomised A/B tests for each new version rather than trusting offline scores.
Do not let a scoring model grade its own homework forever. Version 12 scored very well on the reward model and still lost engagement, so Meta set a ceiling on that score for later versions.
Keep checking general ability while you tune for one goal. Meta tracked public benchmarks to confirm that tuning for engagement did not break the model's basic skills.
Plan for some versions to fail. Meta reports at least one clear setback among eight post-launch versions and treated it as a source of rules for the next ones.
Caveats
- All results are reported by Meta's own authors in a research report. The listed affiliations are Meta Superintelligence Labs and FAIR at Meta (two authors now list OpenAI and xAI, noting the work was done at Meta). No outside party has confirmed the figures.
- The up to 8.8% breadth gain (version 14) and the 19.4% depth gain (version 13) come from different versions, according to the paper's appendix table, so no single model delivered both. The report's '7 of 8 models showed positive lift' holds for each measure on its own, but the table shows a different version with a negative result on each: version 8 on breadth and version 12 on depth.
- The 59.2% to 84.8% figure is the IFEval benchmark score across pre-launch versions 2 to 7, a general test, not a measure on live Meta characters.
- One sentence in the report's text puts version 2 at about 75% instead of 59.2%; this case follows the abstract and table. The 26.6% to 5.8% figure comes from internal and expert testers with an AI reviewer, not from users.
- The report states two time ranges: development from January 2024 to September 2025 in the introduction, and to April 2025 in the abstract, results and conclusion. This case uses April 2025.
- This is a consumer social product built by Meta on its own model, not an enterprise customer deployment. It is included for the documented method of improving a production model with live tests.
- The report gives no cost or team size, and the sources do not say which share of users see the characters. The paper date is the date printed on the paper (2 March 2026 on arXiv, 3 March 2026 on the paper itself).
Sources
- CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Production
arXiv (Meta Superintelligence Labs and FAIR at Meta) · 2026-03-02 · Research paper · arxiv.org
Summarized in our own words from the sources above. Spotted an error? See corrections.