Airbnb
Airbnb's live note-taking assistant for support agents cuts handling time by 3%
Airbnb built an assistant on a fine-tuned open Mixtral model that writes bullet notes while its customer support agents handle a case, instead of one long summary at the end. Airbnb's authors report that in production it cut average handling time by 3%, and by up to 9% in the most complex cases.
Key figures
Copied exactly from the sources. Copy any figure with its citation.
3%
Reduction in average case handling time versus the earlier end-of-case bulk summary, on production cases (difference-in-differences comparison of pilot and other sites; the paper's results section calls the same 3% agent working time)1
up to 9%
Reduction in agent working time for the most complex cases (3+ agents involved, usually solving time over 100 minutes)1
95.2%
(English), 81.8% (French), 89.4% (Spanish)
Share of agents reporting 'Satisfied' or 'Very Satisfied' with the summary feature in a post-pilot survey of over a thousand agents, by language (English, French, Spanish)1
10%
to 3%
Share of agents' working time spent manually writing notes, before and after the feature rolled out (offline analysis)1
The problem
When a customer support case moves between phone, chat and email, and between several agents, each agent has to read the full history before acting and then write notes at the end. That reading and writing takes time, and poor notes lead to mistakes at handoffs. Airbnb's earlier system wrote one summary when the conversation ended. Agents said that was too late to help them while the case was open.
What they built
Airbnb's team built an assistant that writes short bullet notes as the conversation goes. It uses a Mixtral-8x7B model, an open-weight model from Mistral AI (open-weight means the model files are published, so a company can run and adapt them), fine-tuned on Airbnb's own annotated support data. A second, smaller model decides whether a new message adds anything worth noting, so greetings and filler do not create bullets. When agents edit a bullet, the edit is kept in later updates and is also used to retrain the model.
How they did it
The team judged quality offline on 1,200 sampled production conversations, using a GPT-4o-based judge that they checked against a human-audited sample. They then compared pilot sites with all other sites before and after rollout. GPT-4o scored slightly higher on completeness in the offline test, but the authors say cost, speed and data privacy kept it out of production. They also moved the model to a faster serving setup, and an 8-bit version cut GPU use from two A100s to one with a 1% drop in completeness and truthfulness.
Results
On over 92% of production cases, the authors report a 3% drop in handling time overall (their results section calls it agent working time) and up to 9% for the most complex cases. Agents spent a smaller share of their time writing notes by hand, 10% before and 3% after. In a survey of over a thousand agents, most were satisfied with the feature in all three languages, with French lowest. The authors also report that model notes were more complete than agents' own notes.
What to copy
Start from an agent pain point, and build the correction loop in from day one. Compare pilot sites with a control group, report each language separately, and be clear about which numbers are survey opinion and which are measured time.
Lessons for your project
Fit the tool to how agents work. Agents told Airbnb they needed notes during the conversation, not only at the end, so the team changed when the summary is written.
Let users correct the model and feed the corrections back. Agent edits to the notes were logged and used to retrain it, and the retrained model scored higher in the authors' tests.
Weigh a smaller self-run model against a bigger vendor model on cost, speed and data privacy, not only quality. Airbnb says these limits kept GPT-4o out of production.
Measure on real cases against a control group. Airbnb could not run a clean A/B test, so it compared pilot sites with other sites before and after rollout.
Caveats
- All figures come from a research paper by Airbnb's own employees, presented at EMNLP 2025 (industry track). No outside party has audited them.
- The paper gives no launch date for the feature. The deployed date is shown as 2025, the year of the paper, and the rollout may have been earlier because the paper does not date the three-month-before to one-month-after window. The ACL Anthology page gives the month of publication (November 2025) but no day, so no day is given for the source date.
- The 3% and up to 9% figures come from a difference-in-differences analysis, not a randomised test, because a direct A/B test was not feasible. The paper calls the 3% 'agent working time' in one place and 'handling time' in another.
- The satisfaction scores come from a post-pilot survey about whether the feature helps agents. They are not customer satisfaction. The introduction says scores over 80%.
- The paper also says the results imply multi-million dollar yearly savings. That is an estimate from the authors, not a measured result, and is not used here.
- The cost comparison is the authors' own calculation for one query per second, a 4k-token input and a 512-token output. It is not Airbnb's actual spend.
- This is a different Airbnb deployment from its customer-facing AI support agent, which is covered in a separate case.
Sources
- Incremental Summarization for Customer Support via Progressive Note-Taking and Agent Feedback
ACL Anthology (EMNLP 2025 Industry Track) · 2025-11 · Research paper · aclanthology.org
- Incremental Summarization for Customer Support via Progressive Note-Taking and Agent Feedback
arXiv (authors at Airbnb Inc.) · 2025-10-08 · Research paper · arxiv.org
Summarized in our own words from the sources above. Spotted an error? See corrections.