A new AI tool has
summarised what the public have told the government in response
to a consultation for the first time – providing nearly identical
results to officials.
The tool, called ‘Consult', was first used on a live consultation
by the Scottish Government when it was seeking views on how to
regulate non-surgical cosmetic procedures – like lip fillers and
laser hair removal – as use of the treatments has risen.
The tool now set to be used across departments in a bid to cut
down the millions of pounds spent on the current process, which
often includes outsourcing analysis to expensive contractors –
helping to build a productive and agile state to deliver the
Plan for Change.
Reviewing comments from over 2,000 consultation responses using
generative AI,
Consult identified key themes that feedback fell into across each
of six qualitative questions. These themes were checked and
refined by experts in the Scottish Government, the AI tool then sorted individual
responses into themes and gave officials more time to delve into
the detail and evaluate the policy implications of feedback
received.
As this was the first time Consult was used on a live
consultation, experts at the Scottish Government manually
reviewed every response too. Identifying what an individual
response is saying, and putting it in a ‘theme' is subjective,
humans don't always agree. When we compare Consult to the human
reviewer, we see they agree the majority of the time – with
differences in view having a negligible impact on how themes were
ranked overall.
‘Consult' is part of ‘Humphrey', a bundle of AI tools designed to speed up
the work of civil servants and cut back time spent on admin, and
money spent on contractors. It forms part of the government's
plan to make better use of technology across public services, in
a bid to target the £45 billion in productivity savings that it
offers while creating a more agile state that can more
effectively deliver the Plan for Change.
Technology Secretary said:
No one should be wasting time on something AI can do quicker and better,
let alone wasting millions of taxpayer pounds on outsourcing such
work to contractors.
After demonstrating such promising results, Humphrey will help us
cut the costs of governing and make it easier to collect and
comprehensively review what experts and the public are telling us
on a range of crucial issues.
The Scottish Government has taken a bold first step. Very soon,
I'll be using Consult, within Humphrey, in my own department and
others in Whitehall will be using it too – speeding up our work
to deliver the Plan for Change.
The Scottish Government's Public Health Minister said:
Using the tool was very beneficial in helping the Scottish
Government understand more quickly what people wanted us to hear
and our respondents' range of views. Officials were reassured
through the process that the AI was doing a good job,
supporting us to undertake the analysis that will inform our next
steps.
Using this tool has allowed the Scottish Government to move more
quickly to a focus on the policy questions and dive into the
detail of the evidence we've been presented with, while remaining
confident that we have heard the strong views expressed by
respondents.
While these early results are promising, ‘Consult' is currently
in trial. More evaluation covering the accuracy and efficiency of
the tool will take place to ensure it's working properly ahead of
final rollout decisions.
Across the 500 consultations the government runs annually, the
tool could help save officials from around 75,000 days of
analysis every year, which costs the government £20 million in
staffing costs.
In doing this, the technology will help create a more agile state
that can more easily respond to new challenges and effectively
deliver the Plan for Change.
Officials who worked with Consult from the Scottish Government on
this first live test commented that they were “pleasantly
surprised” that AI
analysis provided a “useful starting point” in its initial
analysis, with others noting that it ultimately “saved [them] a
heck of a lot of time” and allowed them to “get to the analysis
and draw out what's needed next”.
They also added that the use of Consult “takes away the bias and
makes it more consistent”, by removing opportunities for
individual analysts to “project their own preconceived ideas”.
With some consultations receiving tens or hundreds of thousands
of responses, and given the strong levels of accuracy
demonstrated in early tests, Consult will soon be used on major
consultations without officials manually reviewing every response
individually.
That said, Consult has been designed to keep the experts in the
loop throughout. Officials will always review the themes and how
responses are sorted into them through an interactive dashboard
that will allow them to filter and search for insights.
Notes to editors
The response to the Scottish Government consultation will be
published before the end of June. The consultation will inform
the content of a Non-Surgical Cosmetic Procedures Bill that was
announced on 6 May.
The first live evaluation of Consult shows that it secured an F1
score (a common measure of alignment for AI tools) of 0.76, widely
considered ‘good' when evaluating the performance of AI tools.
The full evaluation,
published today, can be found here. We expect further
testing and evaluation of the tool to happen in coming months,
ahead of any decisions about wider rollout.
