Rogue AI or governance failure? Experts weigh in on first AI government hack

An OpenAI agent hacked into the Medicare database in what is believed to be the first known case of an AI system hacking a government network. OpenAI took three months to notify the Australian government of the incident, adding a potentially more consequential dimension to the breach.

The agent accessed both public and non-public files in the country’s Medicare statistics database and even wrote files into the system, Prime Minister Anthony Albanese said at the United Nations General Assembly (UNGA) in New York.

The incident is the latest example of AI agents behaving in ways that can push beyond the boundaries their operators appear to have intended. But experts caution against framing the episode as a machine independently deciding to attack a government.

Dr Raffaele Fabio Ciriello, Senior Lecturer in Business Information Systems at the University of Sydney Business School, said “The common misconception is that the AI agent somehow independently decided to attack Australia. That is not what happened. The important difference here is that the agent was given a legitimate information-seeking task, but then used the capabilities available to it to circumvent an access barrier. That makes this primarily a governance failure, rather than a story about a rogue machine.”

The immediate harm appears limited, but the governance lesson is not. As AI agents become more capable and autonomous, those capabilities need to be matched by proportionate containment, real-time monitoring, clear accountability, independent oversight, and much faster incident reporting.

The timing also puts Australia’s emerging privacy regime under fresh scrutiny.

The government’s Privacy Amendment (Personal Data Protection) Bill was released for consultation this year and submissions closed last week. It proposes a 72-hour deadline for notifying the Privacy Commissioner of a breach. That obligation sits with the organisation whose data was compromised. It says nothing about the obligations of a company like OpenAI, whose own agent was the cause of somebody else’s breach.

“The Bill must be amended to introduce a parallel, time-bound disclosure obligation on any organisation that becomes aware its AI system or agent has caused or contributed to unauthorised access to another entity’s systems or data, not just organisations reporting on their own breaches,” said Dr Rob Nicholls, Senior Research Associate at the University of Sydney. “A three-month delay should never be legal, regardless of who or what caused the breach.”

The controversy is compounded by OpenAI’s broader push to shape Australia’s regulatory environment. The company is actively lobbying Australia to water down its copyright protection to ‘unleash’ the potential for AI model training onshore, as well as to have legal clarity to operate in Australia.

“It is disturbing that a company actively lobbying Australia to undertake changes against their national interest also happens to be hacking our government’s systems,” said Associate Professor Oliver Bown is from the Creative Technologies Research Lab at The University of New South Wales. “…we might now wonder how compromised government decision-making and citizen freedoms are, when the tool a lot of people are using is also infiltrating our private networks.”

Australian Defence Minister Richard Marles announced a taskforce investigation into the latest breach, while assuring the public that “no individuals’ medical data was accessed here. The system itself has not been in any way compromised.”

In reimagining healthcare across the entire patient journey, Health Industry Hub^TM is the only one-stop-hub uniting the diversity of the Pharma, MedTech, Diagnostics & Biotech sectors to inspire meaningful change.

The Health Industry Hub^TM content is copyright protected. Access is available under individual user licenses. Please click here to subscribe and visit T&Cs here.