By Devon Whittle, Australia Director, Global Shield Australia
In June, an OpenAI agent bypassed protections on a Services Australia data portal and accessed non-public Medicare statistical information. OpenAI took almost two months to notice what its agent had done, and another month to tell the Government.
At the Joint Select Committee on AI’s first set of hearings, the Government, AI companies, and civil society all agreed on one thing: this incident, and how it was handled, was unacceptable.
There is also broad agreement on the fix. Just like in aviation, pharmaceuticals, and consumer goods, AI companies should be required to properly monitor their AI systems and notify the government when things go wrong. The EU, California, New York and other jurisdictions have already legislated versions of these requirements.
OpenAI and Anthropic both told the Committee that they supported AI incident reporting in Australia. The Government is consulting on what kind of regime it should put in place. And the Committee’s chairs have written to the Prime Minister recommending that Australia introduce AI incident reporting obligations.
Of course, agreeing that incident reporting is a good thing is easy. The fight will be over the details.
The AI companies have urged Australia to focus on ‘interoperability’ and look to California’s system as an example to emulate. Interoperability makes sense when it comes to matters such as definitions and reporting formats. But it can’t be used as an excuse to just copy and paste a regime from overseas.
As we at Global Shield Australia told the Committee, the substance of Australia’s approach, including when obligations apply, what must be reported, and what reports are used for, must be right for Australia.
Copying California’s approach is also particularly fraught given that even the US federal government is now demanding more. In October, after Anthropic disclosed that one of its AI models had lodged fake US visa applications and sent a false tip to police, the US’s new Super Intelligence Force has said that reporting incidents and fixing harms “is not optional”.
A properly designed AI incident reporting framework would give the Government intelligence on the ever-changing risk landscape, enable faster responses when Australia’s national security is threatened, and provide Australians an avenue for reporting when they are harmed.
To do this, Australia needs to show leadership in the face of pressure from the AI giants, which are readying their own campaigns to influence how Australia’s regime will operate. This means learning lessons from overseas jurisdictions and getting four things right.
Four things Australia’s AI incident reporting regime must get right
First, monitoring and reporting must be mandatory. OpenAI says it is currently combing through its files back to November 2025 to see what else its agents have been up to. Incidents need to be caught and reported as they happen, not eleven months after the fact.
Second, monitoring and reporting must be required across the entire AI lifecycle. Limiting the regime to model training, as the Government’s consultation suggests, is like a drug safety regime that doesn’t apply to medicines once they go on sale. Harm can also occur during testing, internal use, and when a model is deployed to customers. The obligation to monitor for and report incidents should cover all of these stages.
Third, the threshold for what is reportable must be credible and calibrated with existing approaches in other sectors. California’s and New York’s regimes only kick in after 50 deaths or serious injuries, one billion US dollars of property damage, or a small list of specific failures. Neither would have caught the Services Australia incident. The aim should be to capture the near misses and lesser incidents, precisely so that major and catastrophic-level incidents can be prevented.
Fourth, incident reports have to be acted upon. The agency receiving reports needs technical expertise to properly assess them, and powers to act in response. This means powers to compel information from AI companies, share incidents within government, and, in the worst cases, order restrictions or withdrawal of AI systems and training runs.
Australia only found out about the Services Australia incident because OpenAI volunteered to tell us. Deputy Prime Minister Richard Marles said that the Medicare data was “kept behind a fence that the AI agent effectively climbed over”. The next fence might be guarding our power grid or water supply. The next company could also choose to stay quiet.
The coming Australian Standards for AI are an opportunity to fix this. They must make AI incident monitoring and reporting mandatory, and get its key elements right. These are rules Australia needs to write for itself, not just copy from Silicon Valley.
