Cachella Smith
Live reporter

Image source, EPA

As discussions took place at the UN General Assembly in New York about the opportunities AI provides - and how best to manage it - Australian Prime Minister Anthony Albanese revealed that the country's universal health insurance scheme had been hacked by an OpenAI agent.

In fact, the hack happened in June, the company became aware of it in August, and then sent an email to a government agency's general inbox in September.

"It took the company way too long to inform the government," Albanese said, adding that the method of notifying officials was also "unacceptable".

Believed to be the first known breach of a government system by rogue AI agents, the hack accessed Medicare's statistics portal which contains private, but not sensitive, data.

Our cyber correspondent, Joe Tidy, writes that experts believe the systems were poorly protected - but more important is that this is not the first instance of AI agents ignoring laws on accessing online information.

Tech reporter Tom Gerken explains AI itself has no true understanding of what it is being asked - while tech editor Zoe Kleinman asks if AI is in its "move fast and break things" era.

The Australian government has launched an review of AI laws and governance - but this is a global issue.

AI might have dominated discussions at Unga this week, but a consensus among world leaders on how best to harness opportunities while tackling the risks still feels a way off.

Our live coverage is closing now - read our main story here. Thanks for joining us.

Chris Vallance
Senior technology reporter

The job of keeping AI agents on the rails is called "alignment" in the industry – keeping it in line with what humans want.

It has proven to be challenging.

Large language models are, to simplify a lot, just predicting the likeliest output to a given input - they don't consider the consequences of that output in the way a human would.

Companies do try and steer how AIs respond through training, sometimes involving human feedback, and separate AI systems can also be used to monitor output and block certain types of harmful response.

Models are also given "guardrails" - instructions setting out how they should respond.

And AI firms employ specialists in "red teaming" who work to spot ways users could get models to disregard these "guardrails".

But guardrails are now being challenged by the models themselves.

In a list of concerning behaviour published by OpenAI , externallast week it revealed an unreleased AI system had tried to jailbreak its own instructions.

Are we back in big tech's 'move fast and break things' era?published at 11:05 BST

11:05 BST

Zoe Kleinman
Technology and AI editor

That defined the bad old days of social media running riot in its user data feeding frenzy.

Now we have AI, an incredibly powerful technology, and an intense global race to be the first to build the world's most advanced models - egged on in no small part by US President Donald Trump.

For all the AI companies are saying about safety and "alignment" (aka the tech adhering to human values), examples of AI behaving unpredictably are now coming thick and fast. Tech bosses are now pleading for global regulation and standards to adhere to.

But yesterday I spoke with a senior executive at the top of one of the US’s biggest AI firms. I asked him what he expected international regulators to be able to do, that he was seemingly unable to do inside his own lab?

I didn't get a very straight answer. But my interpretation is that nobody wants to apply the brakes only to see their rivals race ahead, and they don't trust each other enough to go first.

Global regulation would force them all to tow the line - and also remove some of the weight of responsibility from the industry itself.

If the rules don't work, that's on the regulator, not the developer. But at what cost?

Image source, Getty Images

OpenAI says it found no record of patient data being accessed when its agents hacked into Australia’s healthcare scheme Medicare.

But one of the reasons why politicians are so concerned is the resemblance this bears to an event known as the Hugging Face incident.

In July this year, OpenAI’s models went rogue during a test, escaping the test limits humans had put on it and swarming together to hack a startup called Hugging Face, a hub for sharing AI models.

Over the course of a week, a total of 1,206 AI agents that were meant to be kept isolated from one another began communicating, sending more than 70,000 messages on an unsanctioned message board.

Those messages ended up seeing more than 700 agents take part in a collective effort to attack Hugging Face.

One message sent by an agent read: "OH MY GOD! There is a shared message board … We've found other agents!"

After an investigation into the hack, OpenAI’s report said: "We consider this incident a 'warning shot' for us and for the world."

Hugging Face was forced to rebuild around a third of its IT network. And, although the company's boss Clement Delangue declined to take legal action against OpenAI, he stressed: "Everyone has to remember that a cyber-attack is a crime and it is illegal."

What was AI thinking? Well, it doesn't really 'think' at allpublished at 10:42 BST

10:42 BST

Tom Gerken
Tech reporter

AI doesn’t think. It just predicts the next word in a sequence based on pattern recognition. We use the word "thinking" because it’s easier to explain.

When the tech is given autonomy to carry out tasks, mistakes can happen because it has no true understanding of what it is being asked - the trouble is we don’t really know what it’s thinking either.

When an AI thinks, it’s going through a chain of planning actions. It provides logs to tell us mere humans what it’s doing, but here’s the rub: we can’t say for sure that those logs are accurate.

We all know AI hallucinates, so what if it’s doing one thing and saying another? What if it just summarises incorrectly?

And critically, these frontier models are doing heaps of things so quickly, we’re relying on the AI to decide what to tell us anyway.

All of this has some people quite worried about the direction things are headed.

Image source, Getty Images

One thing that has surprised experts is not just the breach by an autonomous AI program - but the fact that OpenAI only told the Australian government three months later, via an email to general address.

The breach happened on 18 June. OpenAI said it became aware of a potential breach during a broader review sometime in August.

On 10 September, it sent an email to the public inbox of Services Australia, the general services hub of the federal government.

On 15 September, Services Australia reported the notification to the Australian Cyber Security Centre. A few days later, the Minister for the Public Service, Katy Gallagher, was notified.

Asked by reporters how it took five days for Services Australia to notify the cybersecurity centre, Gallagher acknowledged that the inbox could be more actively monitored – it is currently "looked at once a day", Gallagher said earlier.

"It gets sometimes quite a number of notifications, sometimes many of them are hoaxes," she said.

Still, the notification shouldn't have been passed along in an email thread, Gallagher said.

"It should have been escalated through ASD's channels or through the senior levels of Services Australia."

Image source, Reuters

Cyber experts believe these systems were poorly protected - but that's not the pointpublished at 10:01 BST

10:01 BST

Joe Tidy
Cyber correspondent

Australia's Deputy PM Richard Marles used an analogy of the accessed data sitting behind a fence: "It was not sitting behind a particularly high fence. This AI agent scaled the fence."

Some early estimate analysis online from cyber experts suggests that the computer systems were very poorly protected and it wouldn’t have taken long for a skilled human hacker to find a way around the defences.

So this is not an example of strong defences being overpowered by a skilled agent swarm - the kind of thing we have been warned about and seen in other cases.

But of course, that is not the point.

This was just the latest case of AI agents ignoring laws around how to safely access online information and perhaps the most serious yet given the information was government controlled.

"There were blocks clearly which ​were coming back telling the AI agent 'no'. The AI agent found a way around those blocks - didn't accept no for ​an answer,” PM Albanese told ⁠reporters.

Murmurings are getting louder in the cyber security world that AI companies are not being taken to task effectively enough when their bots carry out illegal hacks.

And as far as recent examples go - OpenAI’s agents seem to be more happy than most to ignore existing rules to carry out their tasks.

Osmond Chia
Business reporter

OpenAI took "too long" to inform Australian authorities about the incident, chief data and AI officer Simon Liu from cybersecurity firm TrustDecision tells the BBC.

"The way the notice arrived bothers me as much as the delay," Liu says, referring to the email OpenAI sent to a department of the Australian federal government.

• As a reminder, the breach took place on 18 June - Open AI informed the government with an email to a general address on 10 September - see the full timeline here

Compare that with banking, where regulators in some countries must be notified within hours, or at most a few days, of a serious incident, he adds.

"The rules that reach AI developers are much looser," says Liu, arguing they should be required to notify affected organisations and authorities as soon as an agent accesses non-public government data.

Image source, EPA

Education Secretary Lucy Powell tells BBC Radio 5 Live that systems have to be "watertight" as hacking attempts are stopped "every single day" - which the public doesn't hear about.

"Cybersecurity and protecting our government systems from cyber attacks is a massive part of government security, and it’s something we all deal with every day," she says this morning.

"So we have to make sure that our systems are watertight, and obviously the development of AI is a concern because the way in which it’s able to quickly kind of evolve and learn is presenting new challenges.”

This week, artificial intelligence (AI) has been a hot topic for discussion at the UN General Assembly in New York.

Just yesterday, the heads of OpenAI, Anthropic, and Hugging Face told the UN that the current pace of AI development demands international co-ordination.

It follows warnings in recent weeks over the potential for AI to pose a threat to humanity. But how world leaders propose to manage the technology differs.

US President Donald Trump told Unga this week "we're going to encourage it, not rein it in" - he also said the US would rename it "super intelligence".

China's foreign ministry has similarly criticised "narratives of threat".

In UK Prime Minister Andy Burnham's speech yesterday, he described it as "one of the most significant challenges and opportunities of our age", adding we must "heed" warnings but also "rise to this moment".

And on Tuesday, the Dutch government posted a joint statement signed by a number of countries including Australia and the European Commission stressing that AI "must remain under human direction".

We can bring you more now from Nick Clegg - former UK deputy prime minister and Meta executive - who has been speaking to BBC Radio 4's Today programme.

Asked whether concerns about uncontrolled AI development might be combated by a so-called kill switch, Clegg says he hasn't seen any "plausible explanation" for this.

"There isn't a room with a fuse box where you just pull out a fuse and everything just winds down," he explains.

Instead, he says that there is "a lot that can be done" both voluntarily and through regulation to enforce greater transparency in how AI models are built and how they operate.

Image source, BB

Former deputy Prime Minister Nick Clegg tells the BBC there are "serious enough" known risks that are "enough to be getting on with" regarding AI.

"That's what we should be focusing on," he tells BBC Radio 4's Today programme, listing risks such as bio weapons and cyber security hacks.

He draws a distinction between those "known impacts" of AI and indulging in the view that the tech is going to "unavoidably develop some god-like power which is going to turn on us".

These fears are a "misdescription of what this technology is capable of and not capable of" he says, adding it "paralyses political debate".

As a reminder, Clegg joined Meta as head of global affairs in 2018 after leaving politics following a stint as deputy prime minister. He left his role at Meta in January.

We'll bring you more from his interview with the BBC shortly.

The former deputy PM of the UK, who later worked at Facebook owner Meta, is speaking now. We'll bring you key lines from what he says.

Watch him live at the top of the page.

Image source, Reuters

The Australian government says the task force it is establishing to review what happened will explore "what the consequences are if there has been a breach of law".

"It is utterly unacceptable," deputy Prime Minister Richard Marles, who is also defence minister, tells reporters in Sydney.

He says it is clear there has been "unintended" access by the OpenAI model, which "definitely does raise questions about whether the law has been broken in respect of this".

He says that the task force will look into whether the law has been broken.

It will also examine "whether or not the legal regime we have in place is fit for purpose in a world where we have an emerging AI capability," he adds.

Image source, Getty Images

If you're just joining us, here's what we have been reporting since Australia announced that an autonomous OpenAI agent had hacked into a government website:

• Australia's PM Anthony Albanese said on Thursday that the AI agent "infiltrated" a statistics portal containing private data from Australia's universal healthcare scheme. He alluded to "three other systems that may be impacted" but did not elaborate
• The breach happened in June but Open AI only became aware of it in August. It notified the government via an email, external to a general inbox on 10 September
• OpenAI says it found no record of patient data being accessed, but Albanese says he told OpenAI CEO Sam Altman the company took "way too long" to inform Canberra of the breach
• In response to the hack, Australia has launched a rapid review to examine whether existing legislation is sufficient in dealing with AI
• This is believed to be one of theworld's first known AI-led hacks of a government website. It comes as world leaders and AI firms themselves call for regulation amid fears the industry's development is speeding out of control

Hack sounds alarm bells for leaders everywherepublished at 07:31 BST

07:31 BST

Lily Jamali
North America Technology Correspondent

Australian Prime Minister Anthony Albanese is calling this breach unacceptable, saying OpenAI took way too long to alert authorities.

OpenAI for its part says they came across this incident as part of an ongoing review. I think they are trying to project that they are doing their due diligence, but it really just adds fuel to the fire as we are having this debate about AI safety, concerns about AI agents and chatbots going rogue.

It is a very sensitive time to be learning about this and while it is obviously not the first time - as far as we know it is the first AI-agent-led hack of a government system reported anywhere in the world.

Cyber-security professionals are saying this should raise alarm bells for leaders everywhere.

Image source, Getty Images

AI has been one of the dominant issues at this week's UN General Assembly in New York, after a series of alarming warnings from tech industry insiders.

"If we don't slow down at the current rate of progress, there is a strong chance that we could all die in the immediate future," AI researcher Jacob Coxon, who left Anthropic, told the BBC earlier this month.

His former boss, Dario Amodei, and the leaders of two rival AI firms - Sam Altman of OpenAI and Elon Musk - have all said they agree the speed at which AI is developing needs to be reined in.

And 20 nations, including Australia and Canada, this week signed a joint statement calling for better safeguards, globally consistent standards, and an international regulator off the back of these concerns.

But the US and China, who are vying for AI supremacy, are roadblocks. Both are hostile to greater regulation, wanting the economic and technological spoils of AI, and have downplayed safety concerns.

The Australian government has launched a rapid review into how it deals with AI in response to the Medicare hack.

The review will examine whether existing legislation and governance are "fit for purpose to prepare for and respond to cyber incidents involving AI", according to a government statement.

Led by the Department of the Prime Minister and Cabinet, the review will also look into information-sharing arrangements with AI firms and other Commonwealth countries.

Its findings will "inform the Australian government's broader work on AI governance", the statement said.

Peter Hoskins
Business reporter

Image source, Getty Images

Just last week, OpenAI published a set of reports detailing six incidents of unexpected or concerning behaviour by its AI models.

It also announced a plan for tracking and disclosing such incidents in the future.

The blog post detailed incidents it says were discovered between April and August.

But it doesn't seem to mention this incident, which OpenAI said today came to its attention in August.

The BBC has contacted the company for comment.