Booking.com
Booking.com tests faster in-house language models in Trip Planner, Smart Filters and listing descriptions
Booking.com says it runs in-house fine-tuned, distilled language models in its AI Trip Planner and Smart Filters, and tested generated accommodation descriptions. In live tests lasting multiple weeks the Trip Planner model gained 2.9% more clicks on recommendation cards than OpenAI GPT-3.5, and generated descriptions drew 1.4% more helpful votes than templates.
Key figures
Copied exactly from the sources. Copy any figure with its citation.
+2.9%
Change in clicks on recommendation cards in AI Trip Planner, A/B test of the in-house distilled model against OpenAI GPT-3.5 on real traffic over multiple weeks1
+1.4%
Change in helpful votes for generated accommodation descriptions vs template-based descriptions, A/B test on real traffic over multiple weeks1
15
x faster
Speed of Smart Filters responses at the 99th percentile after deploying the distilled model with Medusa speed-up heads (the paper gives no baseline)1
10
-20x
Reduction in response time with a small distilled model plus speed-up heads, in the paper's offline experiments on a 1.1B-parameter model (not live traffic)1
What it took
The problem
Booking.com wants AI features that answer in real time. Large language models write well but are slow and expensive to run. The authors cite research showing that even 100-400 milliseconds of extra delay can reduce user engagement. The company also needs descriptions for 3 million accommodations, kept up to date as properties change. Template-based descriptions are repetitive and hard to maintain.
What they built
The paper covers three Booking.com features.
AI Trip Planner is a chat assistant that suggests hotels and destinations. A fine-tuned model pulls structured trip details, such as dates and preferences, out of the conversation and passes them to Booking.com's recommendation systems. Smart Filters lets users type a free-text search and turns it into the usual filters. Generated accommodation descriptions use AI to write property descriptions from the attributes partners supply, replacing fixed templates.
For the live features the paper describes an in-house fine-tuned, distilled language model with a speed-up technique called Medusa. Distillation means a small model learns from the outputs of a larger one. Medusa lets a model predict several words at once.
How they did it
In the offline experiments, a fine-tuned 70-billion-parameter Llama 3.1 model labeled 17,000 extra examples taken from Booking.com's production queries. These were combined with 9,000 human-labeled examples to train a 1.1-billion-parameter TinyLlama student. For descriptions, GPT-4 wrote training examples for 10,000 accommodations and content experts edited 273 of them.
The models run on Amazon SageMaker with auto-scaling. Trip Planner and Smart Filters run in real time. Descriptions run in batches when property data changes.
Results
In A/B tests on real traffic lasting multiple weeks, the Trip Planner model led to 2.9% more clicks on recommendation cards than OpenAI GPT-3.5, and generated descriptions earned 1.4% more helpful votes than the template versions. The paper says Smart Filters responded 15x faster at the 99th percentile after the distilled model with speed-up heads was deployed, without saying what it was compared with.
In its offline experiments, the combination of distillation and Medusa cut response time by 10-20x while keeping the teacher's accuracy, and the speed-up heads cut the cost of generating descriptions by 1.9 and 3.2 times.
What to copy
Choose the metric that matches each feature and test on live users. Set a latency budget at the start for features users wait on. Compare the small model with the large model you would otherwise use, on both quality and cost.
All reported figures
| Measure | Figure | Citation |
|---|---|---|
| Change in clicks on recommendation cards in AI Trip Planner, A/B test of the in-house distilled model against OpenAI GPT-3.5 on real traffic over multiple weeks | +2.9%1 | |
| Change in helpful votes for generated accommodation descriptions vs template-based descriptions, A/B test on real traffic over multiple weeks | +1.4%1 | |
| Speed of Smart Filters responses at the 99th percentile after deploying the distilled model with Medusa speed-up heads (the paper gives no baseline) | 15x faster1 | |
| Reduction in response time with a small distilled model plus speed-up heads, in the paper's offline experiments on a 1.1B-parameter model (not live traffic) | 10-20x1 | |
| Reduction in the cost of generating accommodation descriptions with the speed-up heads (two variants), in the paper's offline experiments (not live traffic) | 1.9 and 3.2 times1 |
Lessons for your project
Speed is a feature. For features that answer in real time, such as search filters, the team tuned for response time as well as quality and tested with live users.
Use a large model to teach a small one. Booking.com used a 70B-parameter model's labels to train a much smaller student, keeping quality while cutting delay.
Run each feature through a live A/B test. All three applications were tested on real traffic over multiple weeks.
Pick the right yardstick per use case: clicks for recommendations, helpful votes for descriptions, and p99 response time for filters.
Caveats
- All figures come from a paper by Booking.com and Amazon Web Services staff at the ACL 2025 industry track. No outside party has confirmed them.
- The paper names only "in-house fine-tuned distilled LLM" as the live model and does not say which base model powers the live features. TinyLlama-1.1B (the student) and Llama-3.1-70B (the teacher) are named only for the paper's offline experiments, so the model is shown as undisclosed.
- The +2.9% and +1.4% results are percentage changes with no baseline values, sample sizes or significance tests. The paper does not say how many weeks the tests ran.
- The 15x figure for Smart Filters is stated without the model or setup it is compared with. The 10-20x and cost figures come from offline experiments on a single machine type, not production traffic.
- For the description feature the paper does not say which model wrote the live text. GPT-4 produced the training examples.
- The paper does not state a country or region for the company, so the region is shown as global.
- The paper gives no go-live dates. The deployed date is the paper's proceedings month (July 2025).
Sources
- Speed Without Sacrifice: Fine-Tuning Language Models with Medusa and Knowledge Distillation in Travel Applications
ACL 2025 Industry Track (Booking.com and Amazon AWS Professional Services authors) · 2025-07 · Research paper · aclanthology.org
Summarized in our own words from the sources above. Spotted an error? See corrections.