Tencent
A Tencent team cuts invented links in advertiser support answers by 92.7% with a tuned open-weight Qwen3-32B
Tencent authors describe an advertising platform that answers advertisers' questions with a retrieval system and a Qwen3-32B model tuned with reinforcement learning to avoid made-up links. In a two-week live test at 50% of traffic, the paper reports a 28.6% higher like rate and a 92.7% lower rate of invalid links than a standard retrieval baseline.
Key figures
Copied exactly from the sources. Copy any figure with its citation.
28.6%
increase
Like rate in a two-week online A/B test at 50% traffic, versus a standard retrieval setup with a DeepSeek-V3 model (0.21% to 0.27%)1
92.7%
reduction
Answers containing invalid or invented URLs in the same online A/B test (0.0041% to 0.0003%)1
2.5
s to 3.1s (+24.0%)
Average first-token response time in the same online A/B test, the cost of the new approach1
72%
relative reduction
Offline hallucination rate on a 3,000-question internal advertising set, tuned model versus the same model fine-tuned without reinforcement learning (0.0047 to 0.0013)1
What it took
Cost, team size and timeline were not disclosed in the sources.
The problem
The Tencent authors describe a large-scale advertising platform with many rules, products and account procedures that change often. Advertisers ask support questions about them. A wrong answer is costly, and the paper singles out invented web links as a risk because they can lead to financial loss, compliance problems and legal exposure. Standard retrieval-based assistants, which look up documents and then write an answer, still produced incomplete or made-up answers.
What they built
Tencent's team combined two ideas. First, a retrieval step that follows relationships between products, rules and procedures (called GraphRAG, a search over a map of linked facts) alongside ordinary keyword and vector search. Second, a Qwen3-32B model that Tencent tuned with reinforcement learning, where the model is rewarded for sticking to the retrieved evidence, following the house style, staying safe and using only valid URLs. Qwen3-32B is an open-weight model, meaning its files are public and can be run on a company's own servers. The service is reachable from an official website sidebar, a mini-program and a standalone site.
How they did it
Tencent kept the knowledge graph small on purpose. It tracked which pieces of content real queries retrieved most often and built the graph from that most-cited subset, so searches stayed fast. The paper's main test snapshot had 4,686 entities and 10,712 relations. The model was first fine-tuned on 1,000 human-annotated dialogues and then trained with reinforcement learning on 1,000 prompts. A smaller Qwen3-4B model reranks search results. The generator is served with vLLM on two NVIDIA H20 GPUs, and the knowledge index runs on Elasticsearch. A post-check step adds 230ms for safety guardrails.
Results
On a 3,000-question internal test set, the tuned model had a 72% lower hallucination rate than the same model fine-tuned without reinforcement learning, and gains in expert-rated accuracy, completeness and safety. In a two-week live test at 50% of traffic against a standard retrieval setup with DeepSeek-V3, the like rate rose 28.6% and invalid URLs fell 92.7%. The paper's two result tables disagree on the dislike rate, so it is not quoted here (see caveats). The price was slower answers: average first-token time went from 2.5s to 3.1s. The authors say the system had served millions of interactions over more than half a year.
What to copy
Define the one error you cannot afford, such as a fabricated link, and give it its own metric and its own reward in training. Run a live split test against a clear baseline, and publish the slowdown next to the gains.
Lessons for your project
Pick the failure that costs most and measure it directly. Tencent tracked invalid links separately because a fabricated URL in an advertiser answer can create compliance and legal risk.
Test on live traffic against a clear baseline. A 50% split for two weeks gave Tencent like and link-error rates under real conditions, not just lab scores.
Report the cost of the fix. First-token time rose from 2.5s to 3.1s, so decide whether your users accept slower answers for fewer errors.
Keep the retrieval index small and driven by real usage. Tencent built its knowledge graph from the most-cited content in production queries to keep search time manageable.
Caveats
- All figures are reported by the paper's authors, who are Tencent employees. The paper is peer reviewed at the ACL 2026 Industry Track, but no outside party audited the results.
- The paper does not name the platform and calls it only a large-scale advertising platform; it does not say the platform is Tencent's own, though all authors are Tencent employees. The users are advertisers asking about product rules, procedures and account management.
- The paper gives no deployment date. The 2026-07 date is the publication month of the proceedings; the system had already run for more than half a year when the paper was written.
- The paper contradicts itself on the dislike rate. Its abstract and Table 3 give a 46.2% decrease (0.26% to 0.14%), while Table 2 gives 0.26% to 0.18% (a 30.8% decrease). This case therefore lists no dislike metric. All the live rates are well under 1% (invalid links: 0.0041% to 0.0003%), so the percentage changes are relative to very small bases.
- The 72% hallucination figure comes from an offline test set, not live traffic. The A/B test baseline was a standard retrieval setup with DeepSeek-V3; the paper does not say whether this was the system in production before.
- The paper reports no cost, team size or build timeline beyond the hardware: 8 and 16 NVIDIA H20 GPUs for training and 2 for serving.
Sources
- Towards Faithful Industrial RAG: A Reinforced Co-adaptation Framework for Advertising QA
ACL 2026 Industry Track (Tencent authors) · 2026-07 · Research paper · aclanthology.org
Summarized in our own words from the sources above. Spotted an error? See corrections.