UK Department for Transport
UK Department for Transport's consultation analysis tool: 90% theme recall in live pilots, 75% without review
The Department for Transport and the Alan Turing Institute built a tool that uses AI to find themes in public consultation answers and label every response, with officials reviewing the themes. In live pilots its first-pass themes reached 90% recall against the human-reviewed set, versus about 75% in a blind test with no human input. The department separately estimates, but has not measured, savings of around 50-70%.
Key figures
Copied exactly from the sources. Copy any figure with its citation.
90%
Themes found by the tool that matched the human-validated themes in live pilots (recall)2
0.93
Theme labelling score (F1, where 1 is perfect) in live pilots, comparing the tool's first pass with the human-adjusted result2
over 92%
Raw agreement between the tool and human reviewers on theme labelling, in both evaluation designs2
75%
Human themes the tool found on its own, in the blind test with no human review (recall)2
What it took
The problem
The Department for Transport (DfT) runs around 55 consultations a year. Each produces free-text answers that must be grouped into themes and counted. Doing this by hand is slow and expensive, and for very large consultations only a sample is sometimes read.
What they built
DfT's AI and Data Science team and the Alan Turing Institute built the Consultation Analysis Tool (CAT). It works in two steps. A group of large language models (AI systems that read and write text) proposes the main themes and rare "golden insights" from the responses. After officials review those themes, the models label every response with the themes it mentions. The version evaluated, v1.0, uses Google's Gemini family of models.
How they did it
Development followed DfT's 2023 research on public attitudes to using AI for consultations, which asked for human oversight. Officials review a random sample of responses and adjust the AI's themes before the labelling step. The team tested the tool in two ways: a blind test on 11 questions against human-made benchmarks, and live pilots in which analysts reviewed the tool's output and the original output was compared to their final version. By the report's date it had been used on four consultations, calls for evidence and calls for ideas, plus one more where officials only cross-checked its themes.
Results
In live pilots the tool's first-pass themes had 90% recall against the set officials finally validated, and its labelling scored 0.93 F1 against the human-adjusted result, with over 92% raw agreement with humans. In the blind test, recall was about 75% and labelling F1 was 0.75. Every response is processed, rather than a sample. Separately, DfT estimates, without measuring it, that the tool saves around 50-70% of the cost and time of a medium-sized consultation, and roughly 15,000 hours so far.
What went wrong
Without human review the tool found about 75% of human themes, and accuracy differed by dataset. Reviewers met overlapping themes and some wrong readings of sentiment, so the department keeps the human theme review in place.
All reported figures
| Measure | Figure | Citation |
|---|---|---|
| Themes found by the tool that matched the human-validated themes in live pilots (recall) | 90%2 | |
| Theme labelling score (F1, where 1 is perfect) in live pilots, comparing the tool's first pass with the human-adjusted result | 0.932 | |
| Raw agreement between the tool and human reviewers on theme labelling, in both evaluation designs | over 92%2 | |
| Human themes the tool found on its own, in the blind test with no human review (recall) | 75%2 | |
| Theme labelling score (F1) in the blind test, where humans had created the themes | 0.752 | |
| Responses analysed in live pilots to date | 200,0002 | |
| Estimated saving on the entire cost and time of a medium-sized consultation (department estimate) | around 50-70%2 | |
| Estimated work saved to date compared with full human analysis (department estimate) | roughly 15,000 hours2 |
Lessons for your project
Keep a person in charge of the themes. The tool's blind recall was 75%, rising to 90% after human review, which the team calls the reason review stays.
Test twice: against a hidden human benchmark, then in live use. The two results (0.75 and 0.93 F1) tell very different stories, so publish both.
Plan review effort by sampling. DfT built a statistical method to estimate how many responses reviewers must read to be about 95% confident no main theme is missed.
Check for bias, not only accuracy. The team tested accuracy across demographic groups and found no evidence of systematic differences.
Budget for fixed costs. Review, survey design, project management and report writing remain, so savings grow with consultation size.
Caveats
- The Department for Transport, which co-built the tool with the Alan Turing Institute, wrote and published the evaluation. It is not independent.
- The 50-70% saving and the 15,000 hours are department estimates built on assumptions about survey costs and social researcher feedback, not measured savings. The report also projects savings of 1.5 to 4 million pounds a year if scaled, which is not counted as a result here.
- Live pilot scores compare the tool with humans who first saw the tool's output and then adjusted it, so they are likely to look better than the blind test (75% recall, 0.75 F1). The 90% recall therefore means the tool's themes largely survived human review, not that it matched independent human analysis.
- Accuracy varied a lot between datasets: F1 on main themes ranged from 0.61 to 1 by question in live pilots. Users also reported overlapping themes and some misread sentiment.
- User satisfaction rests on 15 survey responses. The report says model upgrades and method improvements likely explain higher scores on some datasets.
- The model is named only as the Google Gemini family; other models were tested during development. The deployed date is the report's publication date, as no go-live date is given.
- This is a different tool from the Consult tool built by DSIT and the Incubator for AI, which has its own library case.
Sources
- AI Consultation Analysis Tool evaluation
Department for Transport, GOV.UK · 2025-12-23 · Filing · gov.uk
- AI Consultation Analysis Tool (CAT) v1.0 Evaluation
Department for Transport / The Alan Turing Institute · 2025-12-23 · Research paper · assets.publishing.service.gov.uk
Summarized in our own words from the sources above. Spotted an error? See corrections.