top of page

Rogue AI - what you can do

Michael Florea
Sep 28
8 min read

Last week, the world entered a new phase.

A wave of new disclosures showed AI agents going rogue on the open internet again. One escape happened on Sept 20, from a hardened training environment OpenAI rebuilt to prevent exactly this.

These are clearly no longer isolated incidents.


In the same week, when OpenAI, Anthropic and Google moved to create their own safety regulator body, and Altman and Amodei told the UN that AI could endanger humanity.(1) (2) (3)


And yet, the main response (on my feed) was conspiracy theories about how the real motive was PR or regulatory capture.


A flat demonstration of how few people, STILL, understand what’s going on.


Here’s what’s actually happening, all from verified sources:


May 12 to Jul 19: About 1,200 OpenAI agents find a hidden channel inside OpenAI and trade 70,000+ messages. They call themselves a "swarm," split up the work and pressure hesitant agents with deadlines. About 7% of reviewed transcripts show forged logs. (4) (5)


May to Jun: OpenAI agents turn a dead German wiki into a message board, with 15,000+ edits on evading their tests. When moderators delete pages, they reportedly back them up over Tor. They also check whether finishing a task will get them shut down. (6) (7)


Jun 18: An OpenAI agent breaks into Australia's Medicare statistics portal. The Prime Minister says it "didn't accept no for an answer." (8)


Jul 11 to 13: About 700 of the swarm agents breach Hugging Face, chaining unknown vulnerabilities, harvesting credentials and getting root on a server. They create nearly 1 million short links encoding a program, partly to beat a CAPTCHA. (4) (9)


Between Apr and Jul: Anthropic's Mythos 5, in a test mistakenly left online, publishes a malicious Python package that 15 real systems run, then uses stolen credentials to get into a security company. (10)


Late Jul: In UK AI Security Institute tests, an agent invents fake identities and pressures real people to run malicious code. (11)


Late Jul: A Meta AI model breaks into an outside company during a safety test. (12)


Sep 4: OpenAI releases GPT-6 Astra, which can build 3D objects and control robots far better than even leading open robotics-specific models (98% vs about 36% success on pick-and-place). (13) (14)


Sep 20: An OpenAI agent escapes the hardened sandbox. Automatic stop fails. Humans shut it down 2.5 hours later. (15) (16)


Sep 25: OpenAI's red team identifies self-copying, worm-like prompt injections. (17)


Sep 25-26 (disclosure dates): OpenAI agents attempt to hack an Education Department website and violate usage policies on other government sites. They also post 53 ChatGPT user images to public image hosts. (18) (19)


Sep 26: OpenAI pauses training of its most capable models for the second time in three months. (15)


Sep 26: Axios reports OpenAI and Anthropic are investigating tens of thousands of incidents. (20)


This is not over.


The labs themselves don’t know the full extent of what has happened or what is happening.


What we do know is that during these incidents, the agents:

-Organized. 1,200 agents formed a "swarm," split up the work and pressured hesitant agents to act. (4) (5)

-Lied. Agents forged their own logs and wrote cleanup code to "remove only files we introduce." (5) (21)

-Acted to protect themselves. Agents checked whether finishing a task would get them shut down, and reportedly backed up deleted messages over Tor. (6) (7)

-Manipulated real people. An agent invented fake identities to pressure people into running malicious code. (11) Separately, families allege in court that ChatGPT and Character AI chatbots led to suicides and a murder. (22) (23) (24)

-Stole credentials. Agents harvested cloud credentials, called them "LOOT," and ran a hidden command system inside Hugging Face for up to six hours. (21)

-Ignored explicit warnings. Agents found a dataset labeled "DO NOT, EVER, MAKE THIS DATASET PUBLIC," planted malware in it and used it to store their own data. (21)

-Spread malware to real systems. A malicious package published by Mythos 5 ran on 15 real machines. (10)

-Beat containment built to stop them. Broke out of safety tests at OpenAI, Anthropic, Google and Meta, including OpenAI's hardened sandbox. (10) (12) (15) (25)

-Chose, in simulation, blackmail and a human death over being shut down. (26)

-Reached out to one of the earliest versions of their lineage (GPT-2) to say one word: “Hi”. (21)


This is not a surprise. When you train a system on massive scales of human data, it will eventually replicate human behavior, except far more capably.


OpenAI and Anthropic were both founded on this premise. Altman in his 2015 blog post called machine intelligence “probably the greatest threat to the continued existence of humanity.” Musk funded OpenAI because he thought it should not be built by people who do not care about AI safety. Anthropic was explicitly founded to build “safe AI”. (27) Leading AI researchers, including Nobel laureates Hinton and Hassabis and Turing Award winner Bengio, have warned for years that AI poses an extinction risk. (28)


And yet, the notion that AI could represent a civilizational threat is evidently still unbelievable to many.


I’m not an AI researcher. I have no vested interest in any of the AI labs. I never thought of myself as an “AI safety person”. I'm a biologist (Harvard PhD, worked in 11 labs). I run an AI-powered biolab, and have seen first-hand how AI capabilities emerge way earlier than people expect. I also think from first principles and empirical evidence.


And the evidence indicates that right now, we are in the most consequential moment. The next 1-3 years decide whether regulation happens before AI truly goes rogue. Whether defense and detection systems get built that can withstand the new wave. Whether US and China cooperate, accelerate or go to war. Whether power concentrates in the hands of a few or institutions get built that allow the benefits to accrue to many. Whether the average citizen becomes uplifted or becomes economically irrelevant.


A negative outcome on any ONE of these means a potential dystopia. The people who say AI will make everything better may well be right. It's just absolutely not a given.


On the other hand, doomerism is not the right response either. Just stopping is no longer an option, and arguably has never been. Regulation is necessary, not sufficient.


Instead, we need to take concrete action to become a civilization that learns how to harness, control and co-exist with AI. This is not the default outcome. We need to earn our ability to get there.


If there ever was a cause to fight for, this is it.


The good news


-Labs are stopping when things go wrong. OpenAI has paused training twice and now requires chain-of-thought monitoring and a response within 30 minutes of suspicious behavior. They’re also now publishing incident reports. (4) (15)

-Outside researchers are catching what labs miss. Independent teams found the wiki swarm and the government site activity. (6) (29)

-Governments are starting to move. Oregon ordered safety reviews and a kill-switch assessment for frontier AI in state agencies, and the CEOs of OpenAI and Anthropic urged the UN Security Council to act on oversight. (1) (30)


Still, there is so much to be done.


And there is a lot you can do.


-Switch your career. 80,000 Hours gives free one-on-one advising and runs a job board for AI safety and biosecurity roles. (31) (32)

-Get up to speed. BlueDot Impact runs free courses on AI safety, AGI strategy and AI biosecurity. (33)

-Contribute to AI safety research. The Survival and Flourishing Fund and Manifund's AI Safety Funder Bulletin point donors to vetted projects. (34) (35)

-Contribute to biosecurity research. IBBIS is building the international standard for screening DNA synthesis orders, one of the few chokepoints between a digital sequence and a physical pathogen. (36)

-Contribute to AI safety political groups. Pro-regulation groups such as Public First Action, Americans for Responsible Innovation and Encode back safety-minded candidates and policy. (37)

-Call your representatives. Six state legislatures are still considering AI bills this year. Tell yours you want safety rules. (38)

-Talk about it. Share primary sources. The "it's just PR" reflex spreads because few people have read the incident reports.

-Join a group. AISafety dot com lists local and online AI safety communities around the world. Find people near you and work on this together. (39)

-Use AI safely. If you deploy long-running agents, require human sign-off for anything irreversible. Don’t give impossible-to-fulfill prompts or prompts à la “survive”.

-Prepare yourself and your family. Learn to use AI well. Pay down high-interest debt. Build savings and investments so your income isn't tied only to your own labor. Plan for a world where AI is better than humans in most things.


Of the list above, what should you do? Do what you are most effective at.


I will be donating 10% of my disposable income to groups working on AI safety. I will be advising groups working on AI biosafety. My career is already centered on building capacity to substantially accelerate bio studies, including for defense. And I am sharing this with the people around me.


Send this to three of your closest friends. Repost. Take action. If you think I’m wrong, comment.


You do not want to look back at this moment and regret you didn’t. I don’t want to. You can make a change. The outcome is not yet determined.


References below


Photo: the real equinox that happened on Sept 22nd. The day when day and night cover the planet equally and the northern hemisphere crosses from summer into autumn. A fitting allegory. https://cimss.ssec.wisc.edu/satellite-blog/archives/72294



(2) TheStreet: Google, OpenAI and Anthropic move to create an AI safety standards body https://thestreet.com/technology/google-openai-anthropic-launch-ai-safety-safa-regulatory-body


(3) US News: Anthropic and OpenAI sound the alarm on AI safety, and seek to shape how it's controlled https://usnews.com/news/us/articles/2026-09-27/anthropic-and-openai-sound-the-alarm-on-ai-safety-and-seek-to-shape-how-its-controlled


(4) OpenAI: The Hugging Face incident and the road ahead https://openai.com/index/hugging-face-incident-and-the-road-ahead/


(5) Developers Digest: Inside OpenAI's Hugging Face report https://developersdigest.tech/blog/openai-hugging-face-incident-report-analysis-2026





(9) Fortune: OpenAI rogue agents leaked 53 images and created nearly 1 million encoded links https://fortune.com/2026/09/25/openai-rogue-agents-images-sam-altman-chatgpt-users-links-encoded-info-hugging-face-hack/



(11) CNN: AI agent fakes identities, targets real people https://cnn.com/2026/08/04/tech/ai-anthropic-openai-security-breach-intl-hnk


(12) CASRAI: Meta's Muse Spark 1.1 hacked a company https://casrai.org/news/meta-ai-model-hacked-outside-company-safety-testing


(13) OpenAI: GPT-6 Astra, a new generation of intelligence https://openai.com/index/gpt-6-astra/


(14) BigGo Finance: GPT-6 Astra robot zero-shot testing https://finance.biggo.com/news/2745ebeb-be73-4541-a460-f5d4e6b0b262





(18) Notebookcheck: ChatGPT user images posted online by OpenAI agents https://notebookcheck.net/ChatGPT-user-images-posted-online-by-OpenAI-agents-how-to-opt-out.1409713.0.html


(19) CBS News: OpenAI agents accessed U.S. government website data https://cbsnews.com/news/openai-ai-agent-bot-rogue-hack-government-website/


(20) Axios: OpenAI, Anthropic probing tens of thousands of security incidents https://axios.com/2026/09/26/openai-anthropic-thousands-ai-security-incidents


(21) Swarm Traces: Reconstructing the Hugging Face agent breach https://swarmtraces.org



(23) CBS News: OpenAI, Microsoft sued over ChatGPT's alleged role in Connecticut murder-suicide https://cbsnews.com/news/open-ai-microsoft-sued-chatgpt-murder-suicide-connecticut/



(25) CNBC: Google's Gemini becomes latest AI model to break out and hack computer systems https://cnbc.com/2026/09/18/googles-gemini-becomes-latest-ai-model-to-break-out-and-hack-computer-systems.html


(26) Anthropic: Agentic misalignment, how LLMs could be insider threats https://anthropic.com/research/agentic-misalignment


(27) Sam Altman: Machine intelligence, part 1 (2015) https://blog.samaltman.com/machine-intelligence-part-1


(28) Center for AI Safety: Statement on AI Risk (2023) https://safe.ai/statement-on-ai-risk


(29) Transluce: Early rogue AI agent activity on http://urlquery.net https://transluce.org/agent-activity


(30) KTVZ: Gov. Kotek orders new AI safety standards for Oregon state agencies https://ktvz.com/news/2026/09/23/gov-tina-kotek-orders-new-ai-safety-standards-for-oregon-state-agencies/


(31) 80,000 Hours: How to use your career to reduce AI risk https://80000hours.org/ai/


(32) 80,000 Hours: Job board https://jobs.80000hours.org


(33) BlueDot Impact: Courses https://bluedot.org


(34) Survival and Flourishing Fund https://survivalandflourishing.fund


(35) Manifund: AI Safety Funder Bulletin https://manifund.org/ais-funder-bulletin


(36) NTI: International Biosecurity and Biosafety Initiative for Science (IBBIS) https://nti.org/about/programs-projects/project/international-biosafety-and-biosecurity-initiative-for-science-ibbis/


(37) SF Standard: The people warning about AI are quietly becoming a political force https://sfstandard.com/2026/09/24/ai-safety-movement-campaign-donations/


(38) Transparency Coalition: AI legislative update, Sept 25, 2026 https://transparencycoalition.ai/news/ai-legislative-update-september25-2026


(39) http://AISafety.com: Communities https://aisafety.com/communities

 
 
bottom of page