UK Department for Science, Innovation and Technology
UK government's Consult AI tool sorts 2,000 consultation responses and scores 0.76 against human reviewers
The UK government used Consult, part of its Humphrey AI toolkit, to group the responses to a Scottish Government public consultation into themes. It handled over 2,000 responses across six questions, while officials also reviewed every response by hand. The department reports an F1 score of 0.76 and says the tool is still in trial.
Key figures
Copied exactly from the sources. Copy any figure with its citation.
over 2,000
Consultation responses analysed in the first live use, across six qualitative questions1
0.76
F1 score, a measure of agreement with human reviewers, which the department calls 'good'1
What it took
Cost, team size and timeline were not disclosed in the sources.
The problem
Governments run many public consultations, and reading thousands of free-text answers is slow. The UK government says it runs about 500 consultations a year, and analysis often involves outsourcing to costly contractors. Sorting responses into themes is also subjective, since human reviewers do not always agree.
What they built
Consult is a generative AI tool that the UK government developed within Humphrey, a bundle of AI tools meant to cut civil servants' admin time. It finds the main themes in consultation responses and sorts each response into them. The department says officials will always be able to review the themes and the sorting through an interactive dashboard.
How they did it
Its first live use was a Scottish Government consultation on regulating non-surgical cosmetic procedures for example lip fillers and laser hair removal. Consult proposed themes for each of six qualitative questions. Scottish Government experts checked and refined them, then the tool sorted over 2,000 responses into the themes. Because it was the first live run, officials also reviewed every response by hand and compared the two.
Results
The department reports an F1 score of 0.76 against the human reviewers, a standard measure of agreement that it calls good. It says the AI and the reviewers mostly agreed, and that where they differed, the ranking of themes barely moved. Scottish officials said the output was a useful starting point and saved a lot of time. The tool remains in trial, with further evaluation planned before rollout decisions.
Lessons for your project
Run the tool in parallel with people first. Scottish Government experts reviewed every response by hand, so the team could measure how often AI and humans agreed.
Have experts check and refine the themes the AI proposes, and keep an interactive dashboard so officials can filter and review how responses were sorted.
Report an agreed accuracy measure and publish the evaluation. DSIT published a full evaluation alongside the announcement.
Do not scale savings claims until testing is done. The department says more evaluation of accuracy and efficiency comes before rollout decisions.
Caveats
- The department's announcement is the main source and the government's own tool and metric. Civil Service World, an independent outlet, reports the announcement but repeats the department's figures, so it does not independently confirm them.
- Consult is described as in trial. The department says further evaluation of accuracy and efficiency will happen before final rollout decisions.
- The release projects about 75,000 days of analysis and 20 million pounds in staffing cost across roughly 500 consultations a year. That is a projection, not a measured result, and is not counted as an outcome here.
- The release says AI and human reviewers mostly agreed and that differences had a negligible effect on theme rankings. It gives no time-saved figure for this consultation.
- No model is named. The deployed date is the announcement date; the sources do not give a go-live date.
Sources
- Government-built 'Humphrey' AI tool reviews responses to consultation for first time, in bid to save millions
GOV.UK · 2025-05-14 · Press release · gov.uk
- Civil service AI tool for consultation analysis passes first public test
Civil Service World · 2025-05-14 · News · civilserviceworld.com
Summarized in our own words from the sources above. Spotted an error? See corrections.