Australian Government Digital Transformation Agency
Australian Government trials Microsoft 365 Copilot across 56 agencies: positive views, moderate use
From January to June 2024 the Digital Transformation Agency ran a six-month, government-wide trial of Microsoft 365 Copilot. Its published evaluation found staff were positive and reported time savings on a few tasks, but only about one in three used the tool daily and accuracy checks ate into the gains.
Key figures
Copied exactly from the sources. Copy any figure with its citation.
The problem
The Australian Public Service wanted its staff to try generative AI safely, on tools they already knew. Public chatbots such as ChatGPT had reached millions of users, and the government wanted staff to experiment in a safe, familiar setting. It needed to learn whether an AI assistant would deliver real gains at work before deciding how widely to buy it.
What they built
The Digital Transformation Agency (DTA) ran a government-wide trial of Microsoft 365 Copilot, an AI assistant built into Word, Teams, Outlook and other Microsoft tools. It was announced on 16 November 2023 and ran from 1 January to 30 June 2024. Copilot was already covered by existing government contracts with Microsoft, which made it quick to deploy.
How they did it
Agencies joined at different times, and each handed out licences to staff who nominated themselves. The DTA and the consultancy Nous Group evaluated the trial. Their evidence was three surveys (before, during and after), 24 interviews run by the DTA, 17 focus groups and 8 interviews run by Nous Group, and six internal agency evaluations. Over 2,000 participants contributed.
Results
Staff were positive: 77% were optimistic at the end and 86% wanted to keep Copilot. Use was moderate, though. Only a third used it daily, mostly to summarise and rewrite text in Teams and Word. Of survey respondents, 69% agreed Copilot sped up their tasks and 61% agreed it improved quality. Some groups perceived savings of around an hour a day on a few tasks. The DTA concluded there are clear benefits and also real barriers.
What went wrong
Accuracy limited the gains. The evaluation found checking and editing Copilot's output cancelled part of the savings, and up to 7% of participants said it added time. Excel features disappointed users and Outlook access problems blocked use. According to iTnews, high expectations were largely unmet. The evaluation also raised worries about entry-level jobs, bias towards western norms, and dependence on one vendor.
All reported figures
| Measure | Figure | Citation |
|---|---|---|
| Trial participants who were optimistic about Microsoft 365 Copilot at the end of the trial | 77%2 | |
| Trial participants who wished to continue using Copilot | 86%2 | |
| Trial participants who used Copilot daily | 1 in 32 | |
| Survey respondents who agreed Copilot improved the speed at which they completed tasks | 69%2 | |
| Survey respondents who agreed Copilot improved the quality of their work | 61%2 | |
| Perceived time saved on summarising, first drafts and searching, for staff at APS levels 3 to 6, Executive Level 1 and in ICT roles | around an hour a day2 | |
| Trial participants who reported Copilot added time to activities | Up to 7%2 |
Lessons for your project
Budget for training and tailor it to your own work. In the trial, 75% of people with three or more forms of training felt confident using Copilot, 28 percentage points more than those with one form.
Pick the use cases before you hand out licences. The evaluation found people mostly summarised and rewrote text, and that naming specific use cases could lift use.
Test the tools your people actually use. Copilot worked well in Word and Teams, but Excel and Outlook problems held back use.
Plan for checking. The evaluation says verifying and editing Copilot's output removed some of the time savings, so measure net time saved, not only first-draft speed.
Fix data permissions first. The evaluation warns Copilot can magnify weak data security and information management habits.
Caveats
- The figures are the Digital Transformation Agency's own evaluation, built mostly on self-reported surveys and focus groups. iTnews notes the evaluation rests entirely on self-assessment and over-represents executives.
- Participants nominated themselves and agencies started at different times on different Copilot versions, so the results may not reflect the whole public service.
- Scale figures differ. The DTA article says over 7,600 staff in 60+ agencies, the evaluation report says nearly 5,765 licenses across 56 agencies, and iTnews says some 5765 licenses.
- The sources name Microsoft 365 Copilot but not the language model underneath it, so the model is shown as undisclosed.
- The DTA pages could not be fetched directly and were read from web.archive.org copies. The DTA article is dated 23 October 2024. The summary PDF prints no date beyond a 2024 copyright line, so none is given for it.
Sources
- Evaluation of whole-of-government trial into generative AI: Now available
Digital Transformation Agency · 2024-10-23 · Press release · dta.gov.au
- Evaluation of the whole-of-government trial of Microsoft 365 Copilot: Summary of evaluation findings
Digital Transformation Agency · Undated · Filing · digital.gov.au
- Gov Copilot trial unsettled by usage metrics and unmet expectations
iTnews · 2024-10-23 · News · itnews.com.au
Summarized in our own words from the sources above. Spotted an error? See corrections.