Fliggy
Fliggy's Qwen3-32B agent negotiates hotel prices and lifts response rate 12.14 points in a live test
Fliggy, Alibaba's online travel agency, uses a tuned Qwen3-32B model as a business-development agent that phones hotels to negotiate price changes. In a production A/B test on 9,653 real conversations, it beat the earlier rule-driven dialogue system on response rate and task success.
Key figures
Copied exactly from the sources. Copy any figure with its citation.
+12.14 percentage points (p < 0.001)
Share of calls where the hotel responded after the agent's first utterance, production A/B test on 9,653 real conversations versus the earlier intent-driven dialogue system (46.72% to 58.86%)1
+5.94 percentage points (p < 0.001)
Share of price-change tickets successfully negotiated to agreement, same production A/B test (19.32% to 25.26%)1
4.63
(+0.33 over GRPO)
Average dialogue rating (1 to 5) by three human experts on 30 online conversations, versus the same model trained with GRPO, an alternative method1
93.33%
Share of 45 curated bad cases fixed, as judged by three human experts1
What it took
Cost, team size and timeline were not disclosed in the sources.
The problem
Online travel agencies like Fliggy negotiate room prices with hotels every day. A staff member, called a business-development agent, calls a hotel manager and asks for a price change, following a fixed multi-step script. The job needs persuasion over a long conversation, but also strict limits: no over-promising, no made-up facts and consistent numbers across several prices.
What they built
Fliggy turned this job into an LLM agent. It starts from the open-weight Qwen3-32B-Instruct model (open-weight means the model files are public). Fliggy trained it with a method it calls REPO, which rewards the model using three signals: a model trained on human preferences, a second language model acting as judge for softer qualities such as tone and script compliance, and simple rule checks for numbers, format and guardrails.
How they did it
The team built a preference dataset of 6,632 samples, including 252 cases collected from online production, and compared REPO with fine-tuning and other reinforcement-learning methods. The paper says the previous base model, Qwen2.5-32B, made errors in deployment, and business experts gathered 45 such conversations as a bad-case test. The method is designed so that when a new failure shows up after launch, a new rule or judge instruction can be added quickly.
Results
In expert review, REPO reached an average dialogue rating of 4.63 out of 5 on 30 online conversations, 0.33 above the next-best method, and fixed 93.33% of the 45 bad cases. In a production A/B test on 9,653 real conversations against the earlier intent-driven dialogue system, the hotel response rate rose from 46.72% to 58.86% and task success from 19.32% to 25.26%. Both gains were statistically significant (p < 0.001).
What to copy
Test an agent against your live baseline, track a response metric and an outcome metric, and keep a growing set of real failures to retest every new version.
Lessons for your project
Compare against the system you run today. Fliggy tested against its production intent-driven dialogue system, because that was the only other system cleared for live traffic.
Write down the business rules the agent must never break, such as no over-promising and no invented facts, and test them with checks, not only with human review.
Collect real failures from the previous model and turn them into a test set. Fliggy built a 45-conversation bad-case set from errors seen in deployment.
Pick two or three simple live metrics. Here: did the hotel answer, and did the ticket reach agreement.
Caveats
- All figures are reported by the paper's authors, who work at Fliggy (Alibaba). The paper is peer reviewed at the ACL 2026 Industry Track, but no outside party audited the results.
- The paper calls the people on the calls 'customers'; they are hotel managers, so this is a sales-side negotiation with business partners, not consumer support.
- The paper gives no deployment date, test duration or share of traffic. The 2026-07 date is the publication month of the proceedings.
- Only one task is covered, price-change 'chase' tickets. The paper says other trained variants did not meet stability and compliance requirements for live testing, so the live comparison is against the older system only.
- The authors say a task success rate of about 25% is comparable to human business-development staff. This is their own statement and no human baseline is shown.
- The expert evaluation used a small sample: three human experts, 30 online conversations and 45 bad cases. No cost, team size or timeline is given.
Sources
- Teaching LLM to be Persuasive: Reward-Enhanced Policy Optimization for Alignment from Heterogeneous Rewards
ACL 2026 Industry Track (Fliggy/Alibaba authors) · 2026-07 · Research paper · aclanthology.org
Summarized in our own words from the sources above. Spotted an error? See corrections.