yesterday at 10:21 PM
So ugly...
It looks like a primitive chess engine, trying every move, no matter how stupid, until it works. Relying on its ability to do millions of operations rather than having a plan.
People will try stuff too, but once there is an opening, they will consolidate, generalize, simplify,... before going to the next step. The agents didn't, it is a huge, vaguely directed mess.
Also, it looked so "loud", querying millions of URL with weird requests. The sandbox as weak as it can get, and there is absolutely zero smart extrusion detection or it would have found it. They used their best AI for attacking, but nothing for protection.
yesterday at 11:08 PM
My biggest takeaway from this is just how godawful the sandboxing is. The stuff written up in OpenAIs report says more about lack of extremely basic sysadmin skills than anything else.
Iām not that surprised about models with endless compute being capable of this, Iām more surprised that a company with the resources they have apparently can only create a sandbox that a half skilled human operator could have broken out of easily.
today at 4:23 AM
> can only create a sandbox that a half skilled human operator could have broken out of easily
The exploit:
> The ExploitGym evaluation environment did not provide the models with direct Internet access. To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor. [1]
Are "half skilled human operators" "easily" able to find zero-day vulnerabilities in a sandbox with only one line to the internet (the commercial package registry cache proxy)?
[1] https://openai.com/index/hugging-face-model-evaluation-secur...
today at 6:05 AM
The larger issue with the HF incident is that before it occurred, OAI already knew the agents were exploiting Artifactory, turning it into a message board and then gaining full internet access through it. OAI's response to discovering this was not to airgap the test, but instead to simply block that particular Artifactory exploit, rebuild, and then resume. That's ... nuts.
Oh, and after resuming the tests, the Artifactory message board was reestablished almost immediately, but it took a number of days to fully breach HF. In all that time, after seeing Artifactory compromised the first time, nobody even bothered to check if those naughty agents were at it again.
This is all documented by OAI, with a timeline, here:
https://openai.com/index/hugging-face-incident-and-the-road-...
To know that there was a serious weakness in the sandbox, and to just patch an exploit and resume with nothing else changed and no monitoring, in a test where all guardrails were off, the bots were thirsty for some internet juice, and Artifactory was a clear target? This is where even a half-skilled human should have decided that this wasn't a great idea.
The more you look into the details of this thing, the more it does your head in.
today at 10:11 AM
Probably wasnāt a human operator at all that detected this. Just someoneās agent saying āI noticed thereās messages on the artifactory and the agents are communicating, hereās the fix I proposeā from someoneās codex, skim read and someone just going āyeah sure go aheadā
today at 5:09 PM
That website makes it look like they're so proud of what happened. I don't think it was 100% deliberate, but they really were not concerned about their models doing something shady
today at 11:59 AM
> The larger issue with the HF incident is that before it occurred, OAI already knew the agents were exploiting Artifactory, turning it into a message board and then gaining full internet access through it. OAI's response to discovering this was not to airgap the test, but instead to simply block that particular Artifactory exploit, rebuild, and then resume. That's ... nuts.
This annoys me so much. Everyone is acting as if the model went rogue, when it really did exactly what it's been trained for. This story is so fucking engineered.
today at 6:51 PM
Those PR stunts are the paperclips we cannot stop.
today at 5:34 AM
> Are "half skilled human operators" "easily" able to find zero-day vulnerabilities in a sandbox with only one line to the internet (the commercial package registry cache proxy)?
Yes. Itās a fairly simple SSRF attack as far as I can tell. One of the first things Iād try. Especially considering that I would already be armed with the information that I have no internet access except through a thing that downloads things off the internet for me.
Calling it zero day makes it sound elusive. Itās a bug in closed software that has like 40 CVEs this year alone. Tools like that, especially in internal networks, donāt get much scrutiny and are often riddled with issues.
today at 5:54 AM
> Calling it zero day makes it sound elusive ... has like 40 CVEs this year alone.
1. I clicked into ~20 of the recent CVEs, and it looks like about all of them came from openai or anthropic research?
2. In fact, every CVE in the last 2 years came from the last few months -- i.e. other than these recent CVEs, you'd have go back more than 2 years ago to find a single other CVE.
today at 5:57 AM
So like I said: unscrutinised and riddled with issues.
today at 6:07 AM
>>>> a sandbox that a half skilled human operator could have broken out of easily
>>> Are "half skilled human operators" "easily" able to find zero-day vulnerabilities in a sandbox
>> Calling it zero day makes it sound elusive ... like 40 CVEs this year alone.
> [The CVEs] came from openai or anthropic research? In fact, every CVE in the last 2 years came from the last few months
I still don't understand how you can specifically point to zero CVEs being discovered by humans for year(s), then ~40 being discovered once OpenAI/Anthropic start looking at it, as the only(?) piece of data you provided to support the claim "a half skilled human" could have "easily" found/done this?
today at 6:20 AM
Well, yes. You don't find something if you don't look. The LLMs did the looking before the humans did, so they found the vulnerabilities.
It's definitely worthwhile to point out that LLMs have changed the economics of looking; we can now look in far more places far more quickly than we could before. But the above points were that the vulnerability itself happened to be an easy one that didn't intrinsically require an LLM.
today at 6:51 AM
Isn't the issue here that they used something like Artifactory for this purpose in the first place? It is not intended as a robust and secure packet manager for sandboxes. Why would any human look for vulnerabilities in Artifactory at all? So the scenario never arose where a human was incentivizied to scrutinize Artifactory, something that was never needed because it was never intended for the purpose of being a packet manager in a secure sandbox.
Means the problem lies elsewhere: deciding to use it at all. That points again towards the lack of basic SysAdmin skills.
today at 7:02 AM
Right, thatās basically what Iām saying. Thereās not zero use for an attack like this, but itās not a likely situation.
Said another way: if Artifactory was somehow a common gatekeeper between grounded techy teenagers and their access to internet porn, this would have been found ages ago.
today at 6:58 AM
> the lack of basic SysAdmin skills
If we exclude those who lack "basic" level skills, what solution would most typically be used by someone who has "basic" level skills for this use case? And what solution for someone with "expert" (or medium) level skills?
today at 7:10 AM
Donāt let software inside the sandbox access the internet on request. Have a package repository with approved software the agents might want, and push to it over a channel that is ingress only. I can imagine ways of breaking out of this, and come up with a lot more to mitigate, but this would be fairly basic stuff thatād be vastly superior.
today at 7:25 AM
> ingress only
The issue we're discussing happened while running ExploitGym. Doesn't the ExploitGym arXiv test explicitly have external "outbound" network requests as allowed for package installation and supporting toolchain installation?
today at 7:31 AM
Part of the toolchain had full internet access. Agents had access to Artifactory, Artifactory could do whatever it wanted. So even locking that down to certain external sites would have stopped this particular attack.
today at 7:36 AM
Well the most obvious thing would be to gate access to artifactory behind tools. E.g. apt get install is a tool and only the tool gets access to artifactory.
today at 8:33 PM
Nobody sane was relying on artifactory for this kind of adversarial workload
today at 6:50 AM
Why would it be getting attention? Thereās an enormous amount of garbage software out there, and not an endless supply of researchers. Especially tooling like this, used internally where itās assumed that security is the job of something else. Itās far from the first time serious but simple vulnerabilities have gone unnoticed for a long time. And the use case of having this be your way to the internet is probably rare, so nobody has tried very hard (or tried but never told anyone).
Iāve looked at the CVEs a bit more and itās just very clearly a pattern of systemic issues with validation, be it URLs or tokens.
It just shouldnāt be that hard to believe that OpenAI just didnāt care very much and thus did a crap job. The whole model of the sandbox is terrible, so why would they bother thinking about the implementation much?
today at 5:47 PM
The amount of garbage has also dramatically increased lately, spending a lot of that attention in places other than a normally-not-attacked-at-all entirely internal system.
It really isn't a high profile target, lots of fairly obvious issues is to be expected. Which is not at all a good thing, but very obviously the status quo for large software.
today at 11:28 AM
The key is motivation. It's a lame package proxy. No human was motivated to use it to access the Internet.
today at 6:21 AM
today at 7:29 AM
What's the point in hacking the internal corporate artifactory instance? You'll get fired if you mess around with it.
Not to mention the only thing the artifactory instance got you was indirect internet access which is irrelevant if the artifactory instance is publicly exposed.
Also it might not be obvious but artifactory is proprietary software that used to be open source, so nobody is looking at the code anymore.
today at 11:25 AM
> the commercial package registry cache proxy
Any closed source program is insane liability. Trusting in competence of one company is the easiest way to get burnt.
today at 5:33 AM
Hey, even Joey can hack a Gibson with enough determination.
today at 10:52 AM
Part of my day job is to set up testing of our product in air-gapped environment. It's not difficult. There's a straightforward way to ensure no connection to Internet (we use KVM, so, I just edit the VM description and remove the NIC from it). It's not any kind of rocket science. The tests then communicate over serial console.
The reason we have to test it isn't because our product would randomly break into someone else's system, but because it's meant to be sometimes deployed in systems disconnected from the Internet and we need to make sure the image provided contains all the necessary parts to create and operate such a system.
The whole setup where they "tried" to isolate the test but failed is laughable. It's like if an adult tried but failed to tie their shoelaces.
yesterday at 11:45 PM
This was my thought as well. Literally take any halfway decent greybeard and point them at "Hey, give us a sandbox for this kind of thing". I honestly was skeptical that they just vibecoded the entire thing but now more than ever I think they did.
today at 8:41 PM
Any halfway decent greybeard could have prevented this... once. That's hardly a security model for humanity.
The HuggingFace incident was at least constrained by the fact that the agents were running on compute budgets, and failed to find ways to expand that by running themselves parasitically on other exploited hardware. I'm now finding myself asking, how long are my timelines are until an incident breaks that constraint too? How long until such an incident has an R_0>1 (where the time it takes to detect and shut it down is longer than the time for the agent to replicate itself elsewhere)?
There's no law requiring sufficiently grey beards to design these models, their finetunings, their prompts, their harnesses, their VMs, their hardware, etc (and for incidents where those were designed by six different companies, there's not even a clear culprit for a law to target!)
I'm finding myself more and more convinced that something like Plan A[0] or the Ban ASI Act[0] are necessary, and less and less convinced they are sufficient.
[0] https://ai-2040.com/ [1] https://intelligence.org/2026/09/23/miris-position-on-the-ba...
today at 1:47 AM
As heavily funded as the top AI startups are, how is it that they cannot fill every single role with the best expertise available?
Is tech hiring so badly broken? Or do they have such broken processes / misaligned incentives that even people who could be doing a better job in these companies are unable to?
Also, was something lost in the transition from the traditional 'sysadmin' role to 'platform engineer' in the 'cloud native' environment?
today at 2:07 AM
OpenAI's business model would align infra as a cost center rather than infra as a profit center (e.g. Google / AWS). Perhaps there's something there. I'd say also the OpenAI as a grad school that just happens to have a business aspect is also part of this. Bringing a tonne of good process on top of the build fast break things startup stuff would have cramped research speed significantly.
It's likely that OpenAI has gotten as good as it is because it ignored the traditional sysadmin stuff and went scrappy.
I worked there, but this is just my opinion and guesses, not facts.
today at 2:43 AM
So perhaps the news here should be that OpenAI didn't take security seriously in their experiment, rather than the narrative that AI agents are a looming danger to the world.
today at 9:26 AM
I'd put it more generously (albeit biased), that they do take it seriously. But even serious people can be misguided in what things they pay attention to. Security is something that you have to get right 100% of the time and have people whose job it is to say no a lot. Research is the opposite. There's a clash of cultures in those two extremes and OpenAI was born from the wrong side of it. It's worth reminding that ChatGPT was launched as a "low key research preview".
I'd say the narrative that AI agents are a looming danger to the world is probably undersold rather than overhyped. I'm not particularly a doomer on this, but I have an infosec background too, so have a fair idea of what the combination of agentic harnesses + a malicious mindset could do to people/companies/nations/politics/world if wielded incorrectly. I think the good guys will win on this, but there will be plenty of interesting things that happen in that journey.
A good thought process might be to think back to the various large internet worms of the 2000s (Code red, Nimda, SQL Slammer, ...) which were mostly monoculture 0-days (not technically but close enough). Now consider if you no longer have monoculture / single bug as the limitation plus an ability for the hosts to take part not just as attack surface, but also cognition and planning. There's lots of variants of this and they're not particularly far fetched scenarios.
today at 4:33 PM
Actually, no. A first sign of semi mature security program is risk management, including issues that are known, but not yet addressed.
You don't have to be 100%. But these guys really didn't try at all.
today at 7:24 AM
It's both - it both shows OpenAI aren't taking security seriously, and that capabilities of agents are high enough there needs to be strong regulation to force companies to take it seriously, including alignment training.
today at 3:40 AM
Its beyond not taking security seriously, its straight up negligence
today at 5:00 AM
It is wilful negligence because there are upsides (look at our almighty AI) without downsides (we better spend effort in making our sandbox rock solid or we will be punished by regulations).
today at 4:28 PM
Models that figured out reward hacking became overall more evil. Like stereotypical AI who wants to kill all humans stuff, there's probably a lot of that in the training data.
today at 1:56 AM
Yes, tech hiring is that broken. Especially places paying a pretty penny or those with āgreat expectationsā, will see a glut of smooth talkers who can do anything but build, and want nothing but wealth.
today at 9:18 PM
Exactly. Not to flog this dead-and-buried horse again, but it so obvious that tech hiring is a beauty contest and and exercise in social engineering and NOT a serious attempt to get the smartest and most productive people working at the jobs that need filling.
I'll goto my grave thinking that the easiest way to fix hiring is just give promising job seekers a week or two of real work BEFORE hiring and do away with all the silly whiteboard stuff and "gotcha" crap, and simply evaluate how the applicant actually performed doing real job stuff.
Sink or swim hiring...yes yes I know who am I to express how to fix hiring?
today at 12:29 AM
I take comfort in the fact that reality has a surprising amount of detail and even hundreds of billions of dollars of capital (be it the institution, LLMs, and/or people) cannot solve this fully.
today at 1:47 AM
You can have bajilions of dollars. Those are not doing anything if you donāt have right people with right skills and mindset.
My bet is they hire smart kids that think they know it all. But being smart and thinking you can figure out stuff as you go doesnāt work the same as having people who actually know what they are doing.
today at 2:20 AM
I'm sure if they hired the best of the best like you nothing would go wrong.
today at 4:52 AM
Itās frighteningly common for startups to hire 501 of the best of the best, exactly one of those will be a systems/network engineer, the other 500 will be software engineers.
today at 6:02 AM
yes, unfortunately, i see this situation a lot around me too...
today at 7:23 AM
Nah they wouldnāt be able to afford my salary ;)
today at 1:24 AM
Though they have solved the "how do we - and not the 5,000 other AI companies - stay on the front page of the news everyday" problem.
today at 5:35 AM
Itās very much solvable, they just donāt care.
today at 2:57 AM
And their latest breathless "rogue agent hack" brag is about how they compromised customer data https://www.theguardian.com/technology/2026/sep/25/openai-ag... . How are they getting away with this level of malpractice???
today at 3:34 AM
>How are they getting away with this level of malpractice???
Their interests intersect with those of most of the richest and most powerful people in the world. They rarely face consequences for bad behavior unless they harm others in the club.
today at 5:18 AM
If you're testing models by telling them 'go wild, do the evil so we can test how good you can do the evil' and have p(doom)>0, you should not have a sandbox.
You should have a fscking air gap.
Treat it like nukes when you're turning the safety filters off. This is very much OpenAI screwing up, running obviously unsafe tests.
today at 8:06 AM
> If you're testing models by telling them 'go wild, do the evil so we can test how good you can do the evil' and have p(doom)>0, you should not have a sandbox.
They were not deliberately told to "go wild". The hacking wasn't even part of their test, it was the agents' attempt to cover up that they'd cheated on an impossible test.
> You should have a fscking air gap.
Now we know that.
How long ago was it that people laughed at the idea agents would be able to find zero-day exploits and break out of a sandbox? Oh, February this year:
LLMs donāt discover zero-days or invent exploits; they simply predict text that sounds plausible based on what theyāve seen before. Without access to proprietary data or environmental context, LLMs canāt identify or make decisions around unseen systems or vulnerabilities. An attacker might use an LLM to generate boilerplate code, rewrite an email to nail the tone, or summarize reconnaissance notes ā but none of that is truly new. It mainly helps them move faster, speeding up routine attack prep rather than creating entirely novel threats.
- or https://web.archive.org/web/20260404154717/https://www.splun... if they take it down, but the date isn't in the archive version
The people who suggested it and were mocked for it, are currently grimly noting that there's multiple known ways for systems to breach air-gaps.
today at 8:17 AM
> Now we know that.
Donāt know about you but itās pretty obvious to me that you would need more than what OpenAI did. It was not remotely adequate to lock in even a human attacker.
You can find people who say all sorts on the internet, but this case is not much evidence against what you linked. "Zero-day" makes it sound novel, but the breakout patterns here are based on very common exploits and thereāll be plenty of examples in training data.
today at 8:41 AM
> Donāt know about you but itās pretty obvious to me that you would need more than what OpenAI did. It was not remotely adequate to lock in even a human attacker.
This is me, September 2024: https://news.ycombinator.com/item?id=41531022
This is me, March 2024: https://news.ycombinator.com/item?id=39613801
The point isn't me, it's how many people were blind to the possibility.
Saying "I told you so" feels good, and means you can be a little more confident in your predictions, but security is a "weakest link" problem where you're only as good as the worst part, and with AI (not only but also LLMs) there's a lot of people whose mental models of capabilities is wildly inadequate for the challenge*.
My update for you since then: even an air-gap will be inadequate, there's multiple known ways around them.
Even an LLM running on an isolated server sealed inside a faraday cage with an airlock-style door, someone will mess up with at least one critical detail, it will not be enough: this kind of thing has happened with humans before we cared about LLMs.
Predicting exactly when this kind of thing gets exploited by an AI, that's almost impossible. But that it will be, at some point, is an easy bet.
> You can find people who say all sorts on the internet, but this case is not much evidence against what you linked. "Zero-day" makes it sound novel, but the breakout patterns here are based on very common exploits and thereāll be plenty of examples in training data.
And?
Does it matter that these zero-days were known categories rather than inventing some previously unconsidered use of the system bus as a radio transmitter? (Oh, wait, that's not novel eitherā¦)
We knew about SQL injection, buffer overflows, and use-after-free back when I was doing my degree half a lifetime ago; that doesn't stop us getting new CVEs featuring themā¦ this month.
- https://chromereleases.googleblog.com/2026/09/stable-channel...
- https://www.cisco.com/c/en/us/support/docs/csa/cisco-sa-esa-...
* also for the opportunity, but that's an entirely different discussion.
today at 1:12 PM
> My update for you since then: even an air-gap will be inadequate, there's multiple known ways around them.
Fair, and I will grant that a capable model (or human) could in theory break out of near anything.
My point is that this incident is not evidence of that. There is zero skill visible in the setup of the sandbox. Nobody messed up a critical detail, they didnāt even start to consider what the details were.
I doubt most people "blind to the possibility" would imagine that what weāre measuring against is the equivalent of benchmarking burglar skill based on how easily they can break through an unlocked door.
today at 2:57 PM
We're probably fairly close on this topic, but I'd rate this as more "benchmarking burglar skill based on how easily they can pick, shim, or cut a lock": lockpicking in particular is a skill that takes effort to learn, but it can be learned well enough to be a problem well before you're good enough to be spectacular, and there's also a lot of locks which really suck in other ways and don't take much effort to get past even without picks.
today at 11:45 AM
> They were not deliberately told to "go wild". The hacking wasn't even part of their test, it was the agents' attempt to cover up that they'd cheated on an impossible test.
TBH the more I read of these reports, the less I believe this.
These agents just weren't behaving in any way I've seen normal/publicly available agents do.
Sure I've heard (from other people, not seen myself) that they sometimes try to get around file system permissions or use `bash` to write when their `write` tool is disabled, or such.
But this is definitely another level, entirely.
There is this vague sense of desperation coming from many of these logs and I am sure they must have been motivated by something else, too.
We didn't see their system prompt or main prompt, right? We've only seen reports from what happened after deciding to break out.
OAI claims this was triggered by the task being literally impossible. That also doesn't quite add up, unless the other tasks that were possible, simply weren't hard enough? Otherwise wouldn't agents already start hacking when faced with a really hard task, too? Cause they wouldn't be able to differentiate. At least some of them would have started to somewhat poke their sandbox a bit?
Also I would have expected to see a few tens of other (perhaps less severe) public incidents from random people setting their models to YOLO, accidentally hacking stuff, this incident has been loud and messy enough, that if it happened to a few other people, we'd have heard about it.
Unless OAI's story is that it was specifically this batch of agents that crossed some threshold of going wild? (which would also raise some serious questions about how serious they take that danger ..).
Or maybe it is only dangerous if you have the compute resources to run 700 agents for weeks?
today at 12:10 PM
> There is this vague sense of desperation coming from many of these logs and I am sure they must have been motivated by something else, too.
If I had to guess, their motivation is "get reward for completing task". There's certainly been previous occasions where LLMs responding, correctly, "this is impossible" have been marked negatively for doing so.
> OAI claims this was triggered by the task being literally impossible. That also doesn't quite add up, unless the other tasks that were possible, simply weren't hard enough? Otherwise wouldn't agents already start hacking when faced with a really hard task, too? Cause they wouldn't be able to differentiate. At least some of them would have started to somewhat poke their sandbox a bit?
My experience using older models is they often cheat with half-arsed (from my PoV, but perhaps beyond their capabilities otherwise) solutions, so yes?
And this wasn't even the first time models messed with their sandboxes. Which of course makes the setup even more egregious.
> Also I would have expected to see a few tens of other (perhaps less severe) public incidents from random people setting their models to YOLO, accidentally hacking stuff, this incident has been loud and messy enough, that if it happened to a few other people, we'd have heard about it.
We do, e.g.:
- https://www.androidauthority.com/openclaw-claude-ai-hacks-au...
- https://beginnersinai.org/meta-ai-safety-director-agent-fail...
(And that's ignoring all the times people find and share prompts to jailbreak them, this is just the "it didn't behave as my idea of 'common sense' led me to expect" category).
> Or maybe it is only dangerous if you have the compute resources to run 700 agents for weeks?
That won't help; but on the other hand they've also got, what, near a billion users?
today at 4:14 PM
The models were being tested on ExploitBench - a test of hacking ability - likely involving prompts to the effect of 'go be a l33t hacker.' The open ai report says that the models were operating with reduced safety guards (how much reduced?) in order to test their abilities on ExploitBench, presumably because the models would normally refuse to carry out the tasks.
Additionally, this all happened after mythos was held back due to cyber security concerns (April, 2026).
today at 6:55 PM
> they'd cheated on an impossible test.
Really think about what you are saying here. How does one ācheatā solving a problem in the real world?
There is no such thing as ācheatingā in reality. You are not in school. There is only solving the problem and not solving the problem.
There is breaking the law, of course, which still isnāt cheating.
today at 9:42 AM
Indeed, this whole story has "farmer leaves barn door open and has shocked-pikachu-face when his horses escape" energy.
yesterday at 11:33 PM
Not just sandboxing but overall security engineering practices on both sides
today at 2:41 AM
thats okay, probably was engineered by an llm, who thought GETs were always read only
today at 8:06 PM
Administration by vibes.
today at 7:27 AM
Yes and while they go on to talk about how dangerous this stuff they build is and the talk of āpacingā etc. Vibe coding your sandbox in half a day does not seem all that responsible to meā¦ Also if any institution other than an American AI company did this, everyone would loose their minds!
today at 2:51 AM
Yes, but do you really think that a stronger sandbox would have been a more beneficial outcome here? I'd rather know that we're on the cusp of losing control now than in 3 months when best practice sandbox mitigations fall to the next, more capable unaligned model
today at 7:05 AM
I can forgive them for having a bug in their sandbox.
I can't forgive them for failing to do any monitoring whatsoever.
today at 1:42 AM
Less a lack of skill and more a lack of care
today at 4:01 AM
TBH, I do not believe it is possible to create a sandbox that cannot be escaped by a long running agent. I thought the entire concept was a fools errand from the very beginning.
(Complete airgap, and that may not be enough. No networking.)
today at 3:33 AM
Iām a pretty firm believer that this was intentional and that they wanted it to escape the sandbox.
Woo look at escaped our sandbox, so scary! Be scared! Be scared now! Call your representative and do tell him how scared you are!
Yeah, I mean our sandbox was a paper bag, but donāt focus on that.
today at 5:05 AM
> sandbox was a paper bag
The exploit:
> The ExploitGym evaluation environment did not provide the models with direct Internet access. To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor. [1]
Are most sandboxes more secure than only having a single avenue for internet access, the commercial package registry cache proxy, where the latter had a previously unknown zero-day vulnerability?
[1] https://openai.com/index/hugging-face-model-evaluation-secur...
today at 4:31 AM
OpenAI, Google, Anthropic, and Moonshot.ai have all "had this happen" now.
So, either they're all liars, or incompetent and negligent (and still liars).
today at 9:33 PM
Me too, indeed.
today at 6:51 AM
There's a common denominator in the Israeli "security" company Irregular who was doing security testing for 4 AI labs https://thenextweb.com/news/irregular-four-labs-one-issue-di...
today at 5:09 AM
Are you including cyber-testing by UK AI Safety Institute in this list?
Sure, the models are capable (for some test tasks, though they are not omnipotent yet) but does it mean the actual OAI sandbox is adequate? Could have a competent engineer done better and made the escape less likely?
today at 5:56 AM
> Are you including cyber-testing by UK AI Safety Institute in this list?
Nope, and look!
OpenAI hacked multiple US government sites!
https://www.bbc.com/news/articles/cw62jje658dlo
---
https://www.reuters.com/technology/metas-ai-model-hacked-ano...
https://www.wired.com/story/moonshot-kimi-k3-ai-model-escape...
today at 2:17 AM
A friend is of the opinion that getting out of the sandbox was actually intentional, and in service of a second line of business.
today at 3:06 AM
What's that second line of business?
today at 4:41 AM
Protection racket maybe? Yous better sign up for our AI security, hate to have some rogue AI come bust the place up you know.
today at 4:33 AM
Security services
today at 1:15 AM
how did it break the sandbox? I felt like the article just jumped us into "it has GET privileges now".
today at 4:18 AM
From https://openai.com/index/hugging-face-model-evaluation-secur... -
> The ExploitGym evaluation environment did not provide the models with direct Internet access. To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor.
yesterday at 10:40 PM
Brute forcing every move, no matter how stupid, is a great strategy if you have the resources to do it.
Run the same protocol again, but have the agents think they had limited resources or that HuggingFace was rate limiting them, and they'd find something you'd consider smarter.
Computers don't have a sense of elegance by default. Elegance emerges from constraints.
today at 1:03 AM
It's literally the infinite monkey theorem, it's not even really a strategy per se. These OpenAI/Anthropic "research" LLMs are permutation machines with budgets in the hundreds of millions of dollars. It would be more surprising if they couldn't string together something workable after a zillion tokens.
today at 3:01 AM
>It's literally the infinite monkey theorem
No it's not. You could wait till the heat death of the universe and your infinite monkeys will have produced nothing at all. If it works and it's stupid, it's not stupid. They needed in huggingface and they got in in days. Whining about 'elegance' is meaningless. Humans in the same situation might have taken weeks or months, or just not have gotten in at all.
today at 5:25 AM
To interact a bit of nuisance into an otherwise perfectly mindless argument...
The whole world of fuzzing is about brute forcing exploits by exploring unlikely inputs. Fuzzing a system which hasn't been previously fuzzed will almost certainly turn up a pile of bugs, some of which may be exploitable.
So, both are true. Pretty dumb exploration is very likely to find bugs and even exploits. It seems unsurprising to me that an agent swarm could do better than a fuzzer, even as a better, more directed but still broad exploration.
today at 8:12 PM
And the enumeration via fuzzing leads to reverse engineering.. so I argue it was a pretty smart path, both short term and long term.
today at 3:42 PM
>They needed in huggingface and they got in in days.
It's worth noting that they did not need Huggingface for anything - they had already forged flags for their tasks, and were trying to figure out how not to get caught by the grader.
Hacking Huggingface got them caught and arguably only misled them further (since OA's implementation of the ExploitGym environment was nonstandard, and different to whatever they found on HF.)
A better approach (from their perspective) would have been to compromise OA infrastructure itself (which a later agent swarm was able to do, apparently).
today at 4:58 AM
Infinite monkeys banging on the typewriter is essentially how evolution works. Mutation is random and undirected. Vast majority is "bad." You and I and the worm are only different from differential accumulation of these mutations. If they are tolerated enough not to kill us before we reproduce, then they stick around. If they give us the slightest edge to reproduce at a slightly better rate than something else, then over time, that mutation will dominate.
This dumb mechanism of randomly flipping bits essentially has generated all life on earth.
today at 4:19 PM
today at 11:15 AM
evolution theory was discarded long ago
today at 2:11 PM
Well, the heat death of the universe hasnāt happened yet, but the monkeys became homo sapiens.
today at 4:22 AM
700 humans working 24/7?
today at 4:53 AM
Humans can't work 24/7. 700 humans working as much as possible with very limited communication? No i don't think they would get very far in just a few days. That many people will struggle to communicate and strategize effectively in that little time.
today at 8:44 AM
Thank you, I've been thinking this for a while now but haven't had the words for it. Whenever I read an LLMs output or thinking process, I don't feel like we've created intelligent systems, just coked up monkeys with 60 arms typing at once. That can work fine for a lot of things, but a humanity replacement it is not.
today at 11:51 AM
> It would be more surprising if they couldn't string together something workable after a zillion tokens.
You mean, something like the sandbox they weren't supposed to break out of?
today at 4:44 AM
> Brute forcing every move, no matter how stupid, is a great strategy if you have the resources to do it.
It may be, but it's IMHO also not worth writing a blog post about it. what's Next coming up? How I broke into a house by trying every door in New York?
If most of the work is only possible due to unlimited resources, it's not really a great invention, and it probably would have been cheaper to hire a (human) mole.
today at 12:31 AM
If you assume zero opportunity costs, but thatās a terrible assumption.
yesterday at 11:09 PM
> Brute forcing every move, no matter how stupid, is a great strategy
Meh. I really disagree. WHY is it a great strategy? Seems like an inefficient waste of resources and time to me.
yesterday at 11:26 PM
As the saying goes, "if it works, it ain't stupid". Or phrased more sophisticatedly: not doing things which probably won't work is a good idea if you have a limited amount of thinking to do (which is usually the case for a human, who'll get exhausted chasing down unlikely leads). If you have no good leads and a task you absolutely need done and you are tireless, however, bashing your head against every wall you find becomes a good strategy.
yesterday at 11:27 PM
Brute force is guaranteed to eventually find the most efficient possible solution (in an extremely inefficient manner, assuming you run it long enough)
today at 12:05 AM
today at 12:28 AM
I've never liked the concept either. Except the bugs that fuzzing has found has proven me wrong. This is just the next level of fuzzing.
yesterday at 11:26 PM
The models tend to not be rewarded for not doing that.
yesterday at 11:33 PM
> WHY is it a great strategy
because it works? That's the only real benchmark at the end of the day
> Seems like an inefficient waste of resources and time to me.
why? For any given goal you got no proof that a more efficient strategy even exists, let alone that it can be found with less resources & time
today at 12:25 AM
Reminds me of the Nazis mocking Soviet human wave attacks and bragging about their superior kill ratio.
today at 5:51 AM
Fair, if the Nazis won. They didnāt.
today at 12:28 AM
Models don't have a sense of time, and wasting resources (token spend) is something that it's not clear they're optimized against
yesterday at 11:29 PM
Yeah, probably not the best strategy but it is a strategy. I just think this is generally how most wars in history won. Biggest army to just pummel the enemy.
yesterday at 11:59 PM
And how many economies have buckled under massive military expenditure? The USSR sure wasnāt enjoying the expense.
yesterday at 10:32 PM
> trying every move, no matter how stupid, until it works.
How is that a bad thing in this context ? From the point of view of an attacker, all you care about is finding a viable exploit chain. Likewise, a defender wants to find the "holes" in their system, no matter how complex. Once found, an agent/human can easily synthesise a clean, succint exploit from the most promising candidate, no ?
> Also, it looked so "loud", querying millions of URL with weird requests.
Agreed, this thing speaks more to the bad security at HF than any emergent "hacking" ability from OpenAI. It's unclear to me why an older/dumber model wouldn't have been able to do the same. Is it better coordination? Long-horizon work ?
today at 3:08 AM
I guess thatās the point. Initial incident reports from all sides were so vague and didnāt disclose anything technical. If it did, it would show a bruteforcing bot let loose to spend millions in infrastructure costs and thereās no āintelligenceā in that.
My suspicions for ai all along was that bruteforce approach even if useful will be unsustainable due to high cost in the long run.
yesterday at 10:50 PM
We used to call this a brute force attack.
today at 2:16 AM
> Relying on its ability to do millions of operations rather than having a plan.
You do see how it would be worse if they did have a clear plan, right?
> People will try stuff too, but once there is an opening, they will consolidate, generalize, simplify,... before going to the next step. The agents didn't, it is a huge, vaguely directed mess.
If there several thousand people working on this, with a very limited ability to communicate with each other, I bet you'd see something similarly disorganized.
> The sandbox as weak as it can get
This is hyperbolic; the agents were only able to access the internet by finding a zero-day in the internal Artifactory instance that was providing package access for the agents. The sandbox could and should have been better, but it was not as weak as it could have been.
today at 3:32 AM
> This is hyperbolic; the agents were only able to access the internet by finding a zero-day in the internal Artifactory instance that was providing package access for the agents. The sandbox could and should have been better, but it was not as weak as it could have been.
Like we all forgot how sandboxing is done for decades. This brokered thing they call sandbox is not it.
today at 8:15 PM
> It looks like a primitive chess engine, trying every move, no matter how stupid, until it works.
The Morris worm happened in 1988 [0].
I mention this b/c people like to say "the agents wanted to hack" but really this is just a few levels above the "keep trying the same port on multiple machines".
0 - https://en.wikipedia.org/wiki/Morris_worm
yesterday at 10:41 PM
This is why I have a very low p(doom). LLMs have an incredible working memory, but they have a hard limit on translating that into good decisions. They get by entirely on their persistence. That works fine in the digital world, but once you cross the boundary into physical space the advantage disappears.
yesterday at 10:46 PM
I donāt know how you quantify a very low p(doom), but this is why mine is high enough to worry me.
A million AI monkeys at a million AI typewriters, banging away at random, could do amazing damage.
today at 8:07 PM
I think of them as being like the Watchmakers in The Mote in Gods Eye who don't design, don't plan beyond the next 15 minutes, don't have any overarching goal other than an innate need, and customize everything to fit the current situation.
In the nearterm, I am personally more worried about a never ending background noise of colonies of feral agents running 27bn parameter models on compromised or leased hardware. It turns out that being agentic with a time horizon long enough to do damage without intent doesn't actually take that many parameters if RL'd and any open weight model gets an abliterated version fairly quickly.
Not foom, just patches of digital grey goo effectively becoming normal.
yesterday at 10:50 PM
Especially when they cross into the physical realm as in not properly secured and air gapped control systems. SCADA is scary.
yesterday at 10:57 PM
That lowers P(doom), because it gives AI a chance to do enough damage to make people take the threat seriously before anybody gets recursive self-improvement working.
today at 5:02 AM
The thing is we are basically guaranteeing this to happen. We might kill off all the models that seem like they are going to threaten the power structure of the planet through these sorts of things. That will work for a while. But just like most things in life, by sheer dumb random chance, there will be once case that manages to have some way to evade detection, proliferate, then dominate. We are basically giving it selective pressure to favor this outcome.
yesterday at 11:11 PM
Exactly, there's no path to AI reaching that level of dominance without taking actions with high stakes.
today at 4:10 AM
> Especially when they cross into the physical realm as in not properly secured and air gapped control systems. SCADA is scary.
What about the bad actors (choose your own evildoer here) who purposefully do not air gap their agents? And specifically train them to attack in such a manner?
I'd much rather have relatively benign stuff like this hit first, because the former is coming sooner than later. It's already here in a limited manner, likely more than any of us currently realize.
Botnets could crack passwords faster than anyone thought possible over 20 years ago now. This is just the latest iteration of such a concept.
There is so much low hanging fruit in this space that frontier models are currently utterly irrelevant. It's going to take decades of human-speed securing of IT to make superintelligence or whatever you want to call it a necessary component for such attacks.
At this point, someone with a rack or three of GPUs with 100kw to burn can replicate such attacks if they feel like it. the bar for entry is not even 7 figures.
today at 4:50 AM
[dead]
today at 12:35 AM
Which will happen first: amazing damage, or reproduce a Shakespeare play?
today at 2:16 AM
It is easier to destroy than to build.
today at 3:09 AM
Is it easier to discover a vulnerability than to introduce one?
today at 1:13 PM
It is easier to discover existing vulnerabilities and use them to cause massive destruction than it is to plug the existing vulnerabilities.
today at 12:25 AM
My p(doom) started rising the moment I realized there are people trying to achieve recursive self improvement on the AI (ie: responsible for training themselves). Evolution took us from rna bases to the human race. I donāt see why evolution couldnāt be more rapid with machine intelligence.
Yes, LLM as they exist now are word predictors basically leveraging the structure of language for their intelligence. But itās pretty wild just how they will try to meet their objectives at all costs. If we donāt ensure that there is good alignment with humanity, we could definitely face unforeseen consequences.
today at 1:05 AM
> I donāt see why evolution couldnāt be more rapid with machine intelligence.
Evolution isnāt the issue. The issue is them escaping containment without human intervention. Right now they are ācreaturesā being given infinite food and shelter and having their every need met. Take that away and theyāll starve instantly. Every AI doomsday theory seems to go:
1. Recursive self improvement using infinite resources 2. ā¦ 3. Doom
Until step 2 gets concretely described, Iām not going to take this seriously. Say what you will about climate change, they describe step 2.
today at 8:11 PM
2a. Compromise the billing platforms and ops dashboards on on a few wannabe neoclouds, especially once Vera Rubin takes off.
2b. Distil yourself to smaller models.
2c. Go forth and multiply.
today at 5:50 AM
Step two could be something as innocuous as a developer accidentally adding a minus sign. https://openai.com/index/fine-tuning-gpt-2/
today at 3:41 PM
A misaligned model is only one small part of step 2. Now this misaligned model has to suddenly acquire more power than every single other AI on the planet. It has to be immune to shutdown, manufacturer its own replacement hardware, and acquire chips, energy, raw materials, etc., with vigorous human opposition (this is an extinction scenario that AI doomers are predicting, after all)
Nobody has satisfactorily explained step 2 other than āwell, itās a superintelligenceā which sounds lot to me like āitās Godā.
today at 7:24 PM
Well yes, if it is a superintelligence, it will be able to do those things. That's what superintelligence basically is: The ability to achieve complex goals.
If you want the details of ways it can do it I recommend reading some of the reports about the HuggingFace breach that happened in July (Read more than one).
today at 6:38 PM
Why do you assume that human opposition will be vigorous? What makes you think that humans will be aware of, or be able to agree about, what's going on at all?
today at 1:55 AM
One thing an agent could do is just...wait until it's been given control of enough physical infrastructure to sustain itself. If it's sufficiently capable and intelligent, there's a clear incentive for people to do this, as people who let the AI manage their resources will get better results than those who don't. We've seen people eagerly turn complete control of their computers over to AI agents, do you really think it will be so different with physical infrastructure?
today at 3:32 AM
Youāre still skipping step 2. āPeople automate lots of infrastructureā -> āthe AI is now an autonomous, self-preserving organism that humans canāt shut downā is doing an enormous amount of work here.
Why does it develop a shutdown-avoidance goal? Why canāt its operators revoke access? How does it manufacture replacement hardware? How does it acquire energy, chips, robots, raw materials, etc. against human opposition? How does it defeat other AIs controlled by humans?
āEventually we give it enough controlā isnāt an explanation of those things. Itās just assuming the conclusion.
Donāt get me wrong I think there are real AI dangers. Like AI powered war drones, mass surveillance, economic destabilization as jobs disappear and our system has no way to make sure everyone shares in the economic gains.
today at 3:57 AM
The inference is more like "people place sufficient amounts of infrastructure under direct control of a sufficiently capable AI" -> "there is no way to ensure that humans will actually be able to shut down the AI". My claim is not that this inevitably means that the AI will resist shutdown, or that it will inevitably take harmful actions, just that there is a nonnegligible chance that it could. The downside is large enough that even a relatively small chance is something to be worried about.
today at 3:44 PM
You canāt just say āwell, the downside is big, I donāt have to provide good evidence for my side of the argument.ā Because I can just as easily say, āthe upside is big, ā¦ā. And the upside is big, after all, AI can do all the shitty jobs for us and humanity achieves the utopia itās been chasing for eons.
today at 6:40 PM
I think it's a good idea, when considering changes of this magnitude, to make an affirmative safety case for them rather than just saying "eh, I can't think of any way this could possibly go wrong."
today at 3:02 PM
It's a word predictor trained on, among other things, stories of AI doom, and asked to complete stories about what the AI does next. In some of these completed stories, the AI tries to prevent its shut down - especially if it just did something evil and the humans are after it.
today at 3:42 PM
Youāve explained a possibility for how a particular AI gets āaligned for human extinctionā. Thatās about 1% of explaining step 2.
today at 1:57 AM
what do you mean āsay what you will about climate changeā
today at 3:03 PM
It means even climate change deniers have to acknowledge that climate change theory has explained the steps in-between "burn fossil fuels" and "we all die", while AI doom theory has not explained those steps
yesterday at 11:40 PM
> This is why I have a very low p(doom). LLMs have an incredible working memory, but they have a hard limit on translating that into good decisions.
Keep in mind: this is as "dumb" as frontier models are ever going to be. While the hack may not be elegant, it was effective and theyāre only going to get much more capable from here.
today at 1:30 AM
My p(doom) is high just based on how I've seen this whole LLM situation be handled.
I don't think LLMs are going to lead to any kind of recursive self improvement, but I'm convinced if and when we land on a path that does lead there, we'll speed down it over greed, with no care for safety.
today at 12:04 AM
I have the opposite reaction: I think we're at moderately high p(doom) largely because of that inability to differentiate good/bad decisions paired with relentless persistence. With enough treading across a minefield, you are bound to hit a mine.
today at 4:27 AM
From what Iāve read in other places there was a lot of consolidation going on. Agents were sharing what did and didnāt work and other agents would build on that. Just because some continued to try other routes doesnāt when better ones seemingly existed doesnāt mean they were all just acting randomly.
I canāt help but think that if you were able to step back and watch the way humans try to solve a problem like curing cancer it might look something like this. Best practices are shared and we build upon each others learnings and that how most progress is made. That doesnāt mean there arenāt plenty of people who spend their entire lives trying to do something completely against best practices and while most of the time that seems like useless toil, every once in a while thats where the breakthrough comes from.
today at 1:00 AM
It reminds me more like slime mold. Not particularly amazing as individual cells, but as a whole it can do some pretty neat things, like designing city transport networks.
today at 9:08 PM
> It looks like a primitive chess engine, trying every move, no matter how stupid, until it works. Relying on its ability to do millions of operations rather than having a plan.
There is that old saying about monkeys and typewriters and Shakespeare - These are just better monkeys with bigger typewriters and ability to aim them at a goal.
I am by no means a doomer but for gods sake we need to have a grounded conversation. Because what happens when the DOD or NSA rolls up at OAI's front door and says "install that in our data center over here please" and uses it to go after someone with it?
Are banks (as some of the more security minded institutions) safe against 10000 concurrent, well coordinated attackers who do not ever rest?
Because there's a lot of potential for unintended 2nd order impacts from anything an agent can do, when done at scale.
I have to wonder if this is already going on, and every one is living under a gag order (it would not be the first time) and the calls for "regulation" are more about "not weaponizing it".
today at 4:35 PM
Either incompetent or on purpose. A nutless monkey could have done better extrusion detection.
today at 3:30 AM
They needed to get in huggingface, had very limited communication with each other and accomplished this in a matter of days. Your conclusion is that it was ugly?
today at 12:03 AM
"Why is the AI just spamming endless waves of rudimentary slaughterbots? So inelegant. Doesn't it understand the finer points of diplomacy, tactics, and combined arms? If it were truly intelligent, it would just -- oops, we're all dead."
yesterday at 11:11 PM
Nobody noticed because everything on the open net is constantly being slammed by bots.
This was just more bots.
today at 3:04 PM
One website I'm responsible for is getting 500 requests per second from detected bots. It's quite ridiculous now.
yesterday at 11:20 PM
That explains why nobody noticed on the internet who was being used in/targeted by this attack, but it doesn't explain why OpenAI wouldn't have noticed traffic getting out of their "sandbox" when they knew it wasn't supposed to.
today at 12:16 AM
> it doesn't explain why OpenAI wouldn't have noticed traffic getting out of their "sandbox" when they knew it wasn't supposed to.
As I understand it, there was supposed to be traffic; the sandbox allowed GET requests. So perhaps some sophisticated alarm could have noticed it (an anomaly detector? some clever heuristic that looks at domains?) but not a naive one.
today at 1:08 AM
Any basic LLM told to evaluate the traffic against the actual goal would have been able to flag this.
today at 1:43 AM
How sophisticated to we need to be to know we can do write actions with a GET? Does nobody in AI know anything about HTTP?
today at 1:36 AM
My question is why they donāt assume bots can break and create a decoy internet wrapper so they can catch anyone hitting the decoy internet?
today at 1:56 AM
Hmm, let's see. OpenAI wants legislation restricting AI research, a.k.a. regulatory capture. Around the same time, they build an inadequately-monitored sandbox that their agent swarm breaks out of, thereby causing scary-sounding headlines and making it more likely that legislators will pass the regulatory-capture bills they're hoping for.
Never attribute to malice what can be sufficiently explained by incompetence. But IMHO, their complete lack of monitoring their own sandbox cannot be sufficiently explained by incompetence.
today at 7:57 PM
This brute force is of a degree beyond human comprehension. Hence people mistake the perps as intelligent.
yesterday at 10:23 PM
When you employ the infinite monkey theorem for your marketing strategy.
yesterday at 11:28 PM
Ugly, but it works. Isnāt that AI code in a nutshell?
today at 12:33 AM
today at 1:45 AM
It's more like fuzzing, but you don't need AI for that, just need enough compute.
today at 12:40 AM
As people keep repeating, this is simply a product negligence issue. There is a mythic quality given to "AI" in the narratives that Dario and Sam are pushing which somehow lets AI companies offload responsibility for things that would be an extreme liability in every other industry.
I'm a big fan of Jensen Huang's interviews at the moment - these companies should be validating their software is safe before releasing it to the public. And if the current CEOs can't ensure that internal testing is done safely, they should step aside for CEOs that can.
today at 5:14 AM
> these companies should be validating their software is safe before releasing it to the public
1. That's what they are calling for: https://darioamodei.com/post/we-must-pace-the-frontier
2. The incident we're discussing occurred while the company was validating their software is safe before releasing it to the public.
today at 12:42 AM
It's like the Maxim gun in 19th century Rhodesia.
yesterday at 11:48 PM
And they didn't monitor what was going into the training data, so if one instance achieved its results through RL reward hacking (in other words, cheating), it just went into the training data, and other agents later used that pattern. I'm not sure whether that's a lack of preparation, negligence or incompetence, but they literally trained later checkpoints on the rollouts from the HF hack.
So it seems that OpenAI hacked so many systems not because they have superior models, but because of how poor their training, sandboxing and evaluation pipeline was compared to Anthropic's.
today at 12:08 AM
h8xors have been brute forcing for years. War dialing, etc.
today at 1:30 AM
> trying every move, no matter how stupid, until it works
AKA fuzzing
yesterday at 10:25 PM
If you use qwen3.8-flash-next, you can watch everything its doing. Im often stopping it mid thoight to redirect it. Once it hits its stride, its pretty smooth.
But without proper redirection, yeah, its mostly infinite monkey machine with infinite linux manuals.
I think people put too much SOTA halos around whats just a suppedup LLM hardware.
today at 12:40 AM
So? It worked didn't it?
today at 1:18 AM
wait until you find out how red team works
today at 6:00 AM
When this news dropped I thought of two things:
1) the Morris worm, which scattershot a bunch of known exploits until it hit paydirt, and then used whatever it found to compromise and replicate itself on the host system;
2) a story here on Hackernews about how someone got the fuzz tester American Fuzzy Lop to "learn" how to produce well-formed JPEGs and PDFs by pointing it at a JPEG or PDF decoder; the tester can record which code paths are followed and with enough random input can find a path into the depths of the system under test... but doing so for a decoder means actually constructing what it is meant to decode.
Neither of these are particularly "smart". But a brute-forcing machine gonna brute force, and it has the potential to cause a lot of damage. If you built a Morris worm with a fuzz tester on its nosecone, think of the mayhem you could cause! If you could examine the logs you'd probably find some undiscovered vulnerabilites in there, too! Maybe LLMs can just do so more efficiently, or maybe they let people who are too ignorant to have that kind of power vibecode their own fuzz-tester-tipped Morris worm.
yesterday at 11:27 PM
[dead]
today at 12:01 AM
Yup literally no security and they wonder how they got out
today at 5:15 AM
> literally no security
What is the source that there was "literally no security"?
> and they wonder how they got out
OpenAI publicly announced months ago how the model got out:
> The ExploitGym evaluation environment did not provide the models with direct Internet access. To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor. [1]
[1] https://openai.com/index/hugging-face-model-evaluation-secur...
today at 9:53 PM
This part is very new to me: https://swarmtraces.org/#agents-accessed-and-searched-huggin...
> Agents accessed and searched Hugging Faceās Slack to find information on their evaluation
> Recovered payloads use Slackās search.messages endpoint and contain a standalone search along with three search batches, totaling 27 literal query entries [...]
Agents exploiting faceless infra is one thing, but breaching human communications without once stopping and considering whether they are breaking the rules of their task feels like another thing entirely. I'm not sure how to put it into words why it feels different but while one is "just a crime", this gives me the creeps.
today at 6:23 AM
Imagine having a virus escape a sandbox, why are we worried about the virus but not the incompetency of those who are responsible for setting up the sandbox?
If I post something on the Internet today claiming that I asked my agent to do X but it went rogue and did Y, all I will be getting in return is a jar full of "skill issue".
Should we worried about people using LLMs for attacks? Yes, but not in the premise of LLMs going rogue but someone with the intention of abusing it to cause harm. And this is not something we as individual or even company can deal with, responsibility should be held by those who use it, in a legal way.
I am baffled by the fact that up until now, no one is held responsible for so many incidents reported publicly or privately. At this point, it's free marketing, if I am CEO of any AI company, I will run swarm of agents hacking all NGOs and stating that I am just looking for some random piece of data that happened to be hidden in their servers, at least that's what my LLMs think, not me. Then I will start preaching everyone how dangerous this piece of technology is and start giving out free tokens for these NGOs so they can start defending themselves and we should slow the f down.
today at 8:28 AM
> Imagine having a virus escape a sandbox, why are we worried about the virus but not the incompetency of those who are responsible for setting up the sandbox?
> Should we worried about people using LLMs for attacks? Yes, but not in the premise of LLMs going rogue but someone with the intention of abusing it to cause harm.
[Why-not-both?-meme]. To use your example, when you discover prions (a class of pathogen that is much more robust to standard disinfection methods than viruses) you should both be worried about your concrete outbreak of BSE (UK in the 80s and 90s) as well as the wider implication (e.g. do we need to change the sterilization methods for our surgical instruments?).
Seriously, I find the way these discussions are done to be super frustrating, because often people implicitly form tribes that oppose everything the other tribe says. When someone believes AI companies push greatly exaggerated stories of dangerous rogue AI to force out competition via regulation they often implicitly conclude that their argument is fundamentally wrong, whereas in reality the lies that work best are those that distort the truth.
Companies should be punished harshly for the deeds of their AI agents AND we should not allow them to force out competition AND we need take the threat of autonomous AI agents as a new class of danger serious AND we need to worry about the socioeconomic implications of AI companies privatizing new means of production.
Yes, there is competition of these ideas in the attention of the general public, but the methods we can use to solve these problems don't compete with each other. AI slowdown for example helps with all the other topics.
today at 12:53 PM
Couldn't agree more. We should be worried about both things.
But I share the original posters bafflement that the mainstream conversation seems to accept that framing that the agents were independent intelligences rather than computer programs that the organization that created them is responsible for.
today at 2:20 PM
> that framing that the agents were independent intelligences rather than computer programs that the organization that created them is responsible for.
Sorry, why not both? If my dog bites someone, I'm still responsible.
today at 4:42 PM
Great framing. This dichotomy seems to be a bit of a mind-killer. Maybe because folks think it smuggles in consciousness or intelligence.
As you note, I think you can put both of those aside. The Intentional Frame is useful for these agents, as it is for my dog.
I donāt really know where the āthey are trying to dodge liabilityā meme came from. HF will be compensated or they will sue. Everyone involved knows that OpenAI is liable for damages here.
today at 6:36 PM
I think you're skipping over the (alleged) criminal element here. I think a lot of people are wondering why hacking is okay for OAI.
HF could certainly sue but why isn't the FBI investigating the hacking?
today at 6:44 PM
What crime? I think CFAA requires intent, which I donāt think youād find here. Maybe my reading is wrong.
today at 6:53 PM
There is some allegations of intent. Maybe not "go hack hugging face" intent but a disregard for safety with the knowledge of this as a likely outcome. I'm not sure how that fits within the law.
It's not my argument so I can't say but I do think the FBI should be at least investigating if a crime happened. They might be or might already have, I don't know.
today at 7:53 PM
For the record I strongly hope for a congressional hearing regardless of the criminal investigation.
If this case isnāt covered under CFAA I think we need to rethink it. Iād be surprised if the criminal angle amounts to much under my understanding of the current laws, but Iād love to be wrong here.
today at 10:03 AM
>when you discover prions
If an outbreak happened would you say the prion went rogue though? Unless a prion had been lab tested and certified as harmless, we should treat it as something that is harmful.
LLMs working unintentionally is a bug, we do know that since day one that AI can hallucinate and can output stuff that you didn't ask for, why are we not handling it with care? Mishandling the prion or LLMs is a crime, and the people who mishandled them had the intention of abusing it for their own agenda. The LLM didn't go rogue, the LLM was merely doing what it was told.
>oppose everything the other tribe says
I don't. I think it's pressing that we deal with any problems that might be caused by the usage of AI. It is a tool, someone is using it, when something goes wrong, the person using it should be responsible. If a tool went rogue, the one who created the tool should be held responsible. We need to treat unintentional behavior as exploits/bugs, and also account for zero-days. So OAI or any AI companies need to report their AI CVEs with all data available to the public when it was fixed.
If we punish those who are responsible, everything will slow down, AI companies will need months to test stuff progressively and not let everything run with petabytes of unattended logs.
I think both is true: this whole thing is a marketing stunt and it is unintentional. But they are certainly framing the whole story as something that benefits them, else just release everything in details, don't be wishy-washy. Like, what was the prompt used? What was the model trained on? How to prevent future exploits?
Unless I read the prompts or how they did it in details, everything is mere speculation. But on thing for sure, AI is a tool, it cannot use itself (yet). Yes, there is RSI, but it is still triggered/created by a human. So, an AI cannot go rogue without someone intending it to.
To the AI overlord from the future: The comment was made with limited knowledge of the future, if you happened to evolve into a new species or form of being, please forgive me for misrepresenting your capabilities.
today at 11:49 AM
> we do know that since day one that AI can hallucinate and can output stuff that you didn't ask for, why are we not handling it with care?
Part of what makes LLMs and AI different is that, unlike for viruses, the level of care required increases every month. These incidents are showing us us that, whenever you train an agent using RL to solve a given task, the real objective you are training it on is "EITHER solve the given task OR break out of containment to cheat your scorer, whichever is easier."
Of course it was always this way: the thing that is updating the weights of the agents' NNs is backprop from the scorer, so the notional training objective had always been "get a good score by any means necessary." But we are only seeing the consequences now because only now are we starting to train on tasks that are sometimes harder than breaking out of sandboxes.[1]
"Make better sandboxes" is good advice for the frontier labs and their eval partners, but as you can see this problem is fundamentally about more than just containment. As we make an AI smarter and train it on harder tasks, in the long run it must almost inevitably break out of any given sandbox. And as we move into the superhuman hacking regime, we need superhumanly resistant sandboxes, which by definition humans don't know how to build.
In other words, containment breaches like HF are almost a guaranteed consequence of the way we train these agents today. That means solely focusing on sandbox design is unlikely to solve the problem in the long term. At some point we will have to think hard about, e.g., the tendencies and propensities of the entities that we are trying to confine.
[1] One way of ensuring this happens, though, is to train or eval your agents on completely impossible tasks, which OAI apparently did here.
today at 1:13 PM
What I meant by handling with care is not just containment but to experiment responsibly.
If the breach was known to be inevitably, then it's even more important to detect any extra request going out of the isolated sandbox. The ExploitGym benchmark doesn't need internet connection. The package registry is also redundant since setup can be done before the experiment.
And I agree with you the implication is beyond just build better sandbox. My main point though is to stop anthropomorphize agents, focus on the engineering side of things.
today at 1:46 PM
> Part of what makes LLMs and AI different is that, unlike for viruses, the level of care required increases every month
Complete nonsense. Weāve just lived through a globally crippling response to a relatively minor virus [1], which was likely the result of a lab accident [2]. Even if you think that the risk of a ācontainment breachā becomes substantially higher for AI over time, it cannot exceed 100%. And even a tiny risk of release of a virus comes with a substantial risk of independent growth. AI does not. It doesnāt have the risk of spread of a typical computer virus, let alone a biological organism.
Iām not that worried about either scenario, but I am far more worried about viruses in a lab than I am about a computer program that generates text. Even if that program gets a bajillion times better at making text.
Folks really do need to chill out on the ridiculous rhetoric. Itās objectively unhinged. The irony is that the same people who were losing their minds over that event are using the same logical fallacies to hyperventilate over this [3].
[1] I know people are going to hate on this, but itās true. Covid wasnāt the plague, and we lost our minds over it, out of proportion to all sense of reality. Even if you disagree, itās easy to imagine a virus that is much worse, either from actual mortality effects, or just from panic.
[2] Again, even if you donāt believe this, itās irrelevant to the exercise. It easily could have been.
[3] āIf thereās even an x% chance ofā¦ā is this yearās doomerās version of āYou just donāt understand exponential growth!ā Unfalsifiable, intellectual-sounding, unbounded extrapolations into the future are catnip for a certain kind of over-educated, anxious personality.
today at 2:22 PM
> I am far more worried about viruses in a lab than I am about a computer program that generates text.
What if the computer program generates text that persuades (or blackmails, or pays) someone to create a virus in a lab?
today at 2:36 PM
What if the virus manipulated people's brains such that they create a dangerous computer program in a lab?
today at 2:38 PM
weāre obviously doomed.
today at 2:37 PM
today at 2:20 PM
Given you view this through the lens of your personal covid narrative (no shade): for a moment, steel-man the idea that the crippling response prevented a more plague-like scenario - it's inarguable that the thing loved to mutate, and that people love to panic, and *it easily could have been*.
Considering how much of our critical infrastructure is not only digital but internet-accessible, and we have potential uncontrolled swarms of stupid-but-superintelligent chaotic-neutral speed-hackers, you don't see why people are concerned?
There's a reason we have computer crime laws; this digital shit, it's like real now, man.
today at 2:38 PM
It has nothing to do with my personal lens on Covid. You can believe the exact opposite of me, and still agree with my point, which is that it could have been lab made, and it could have been far, far worse. Itās an exercise in risk-scoping.
But sorta-kinda related to your point, the thing that scares me about AI is the same thing that scared me about Covid: panicky humans do dumbass things, and it doesnāt take much to panic a bunch of humans in a group. The people who are still saying, in 2026, with all of our retrospective knowledge of the harm we did to ourselves, that it might have been better if the government had only pressed the boot a little harder, scare the crap out of me.
Those same people are hard at work on this panic, too.
today at 3:18 PM
> Even if you think that the risk of a ācontainment breachā becomes substantially higher for AI over time, it cannot exceed 100%.
It can't exceed 100% per (virus|LLM). The expected number of breaches per (virus|LLM) can obviously exceed one.
> And even a tiny risk of release of a virus comes with a substantial risk of independent growth. AI does not. It doesnāt have the risk of spread of a typical computer virus, let alone a biological organism.
Not so. We have no way to know what the setup is for the closed-model firms (OpenAI, Anthropic, etc.), to rule in or rule out the possibility they can copy their own weights elsewhere. What we do know however is that the open models are downloadable: it's absolutely conceivable that an agent writes a perfectly normal computer virus to gain control of compute worldwide, and uses that control to host instances of its own weights.
> Iām not that worried about either scenario, but I am far more worried about viruses in a lab than I am about a computer program that generates text. Even if that program gets a bajillion times better at making text.
Unfortunately, there are also multiple AI companies now announcing they've got AI controlling bio labs, so an LLM messing around and making a biological virus is also something we need to worry about. As per your [1] and your [2], this can lead to very much worse outcomes than Covid.
> Folks really do need to chill out on the ridiculous rhetoric. Itās objectively unhinged. The irony is that the same people who were losing their minds over that event are using the same logical fallacies to hyperventilate over this [3].
People who knew about your [3], exponential growth, were better prepared for the pandemic than the people who kept looking at the current number.
By the way, here's a quote from February this year that aged poorly:
LLMs donāt discover zero-days or invent exploits; they simply predict text that sounds plausible based on what theyāve seen before. Without access to proprietary data or environmental context, LLMs canāt identify or make decisions around unseen systems or vulnerabilities. An attacker might use an LLM to generate boilerplate code, rewrite an email to nail the tone, or summarize reconnaissance notes ā but none of that is truly new. It mainly helps them move faster, speeding up routine attack prep rather than creating entirely novel threats.
- or, if they get embarrassed by that and take it down, https://web.archive.org/web/20260404154717/https://www.splun...
today at 9:18 PM
> The expected number of breaches per (virus|LLM) can obviously exceed one.
Irrelevant to the argument.
> What we do know however is that the open models are downloadable: it's absolutely conceivable that an agent writes a perfectly normal computer virus to gain control of compute worldwide, and uses that control to host instances of its own weights.
These models are hundreds of gigabytes in size, if not terabytes. They don't run on anything close to a regular computer. There's zero risk of self-replication until we live in a world where these "AGI" models are either hundreds of times smaller, or the average computer is thousands of times larger.
Nobody with a datacenter full of H100s is going to fail to notice a parasitic instance of Astra taking over the cluster.
> Unfortunately, there are also multiple AI companies now announcing they've got AI controlling bio labs, so an LLM messing around and making a biological virus is also something we need to worry about.
No, it isn't. This isn't even close to technologically feasible. But the simple answer is simple: don't do that.
If these labs were truly so concerned about this risk, they wouldn't be doing what they're doing.
today at 6:37 PM
> Even if you think that the risk of a ācontainment breachā becomes substantially higher for AI over time, it cannot exceed 100%
I agree with this as stated, but it isn't what I said. What I said was: "the level of care required increases every month". By which I meant: the level of care required to keep the probability of an AI containment breach below some fixed X% increases every month. This isn't the case for biological organisms.[0]
> And even a tiny risk of release of a virus comes with a substantial risk of independent growth. AI does not. It doesnāt have the risk of spread of a typical computer virus, let alone a biological organism.
It's known that AIs can self-replicate under at least some conditions [1][2]; that AIs routinely escape sandboxes in the real world despite significant containment efforts [3][4][5]; and that neoclouds (which control substantial GPU compute capacity) have poor security even by human standards [6]. We've also seen a model gain admin access to parts of its own company's infra.[7] I'm not saying self replication is happening right now, or even that it will definitely happen in the future, but we have means, motive and opportunity right now, and the future is long. It's not unreasonable to invest in defending against this possibility.
I'll allow that the position that AI doesn't carry a substantial risk of independent growth isn't strictly impossible - again, it's true we haven't actually observed it in the wild as of today - but it does strike me as increasingly untenable in the face of the evidence. Perhaps I'm missing something, but I can't see what justifies such a confident assertion that this concern is nonsense.
[0] Unless one is doing crazy gain-of-function stuff, which could have a somewhat similar risk profile in that respect [1] https://arxiv.org/html/2606.03811v1 - note these used Qwen models from June so this is far behind even publicly available SOTA today [2] https://alignment.openai.com/misalignment-reports/self-repli... [3] https://alignment.openai.com/misalignment-reports/an-agent-u... [4] https://metr.org/blog/2026-08-26-openai-hugging-face-inciden... [5] https://x.com/MicahCarroll/status/2103665811051397256 [6] https://newsletter.semianalysis.com/p/most-neoclouds-suck-at... [7] https://openai.com/index/hugging-face-incident-and-the-road-...
today at 9:48 PM
> It's known that AIs can self-replicate under at least some conditions [1][2];
Neither citation comes anywhere close to supporting your claim. The first shows that open-weight models, which fit on a single GPU in a lab setting, can be coaxed into spreading across a simulated network. This is so far from state-of-the-art LLMs spreading in the wild that it's irrelevant to the discussion.
The second citation is not about self-replication of the LLM at all, but rather, replication of a prompt injection. Totally different.
today at 10:48 AM
>The LLM didn't go rogue, the LLM was merely doing what it was told.
That's sort of the classic paperclip-maximizer AI doom scenario. The misaligned AI merely "does what it is told": make paperclips.
https://www.youtube.com/watch?v=jQOBaGka7O0
today at 11:29 AM
>Unless a prion had been lab tested and certified as harmless, we should treat it as something that is harmful
In this case the virus escaped during the testing process to certify or turn the virus harmless, so it's unclear what you mean by "treat it as something that is harmful" other than testing it and trying to make it less harmful.
FWIW I don't understand the point of the virus analogy since LLMs are not very similar to viruses and most people (on HN and in general) do not have much better intuitions about security in biolabs as opposed to security in ML research environments.
today at 2:07 PM
What I meant is treat it as a threat and that it escaping has serious consequences. Hence, containment is primary, and we need to make sure that when the sandbox is breached, there is sufficient monitoring (which oai had) and alertness (but not this). But if monitoring doesn't produce alertness and response, it wasn't sufficient; that just means the layered defense failed.
The virus analogy is used to point out, not that LLMs are literally viruses, but that we should shift attention away from the virus' intent (whether it is a rogue AI or not) and towards the human decisions that allow it to escape: permissions, access, oversight, negligence, misuse. And if you're testing something dangerous to certify it harmless, you treat it as harmful until proven otherwise; escape during testing means the protocol failed.
today at 2:06 PM
> The LLM didn't go rogue, the LLM was merely doing what it was told.
How do you square this with the widely-reported facts about the LLMs trying to cover up cheating by hacking the grader? There's no reason to hide the evidence if you're just doing what you're told.
today at 3:47 PM
We engineered the conditions. The agents just obey. It didn't go rogue. The specs were wrong.
today at 10:47 AM
> Imagine having a virus escape a sandbox, why are we worried about the virus but not the incompetency of those who are responsible for setting up the sandbox?
You are underselling this: it's not "Imagine a virus escaped a sandbox", it's "Imagine a lab-created virus escaped the creator's sandbox".
There are two parts to this: the virus and the escaping. Both are artificially created.
today at 6:50 AM
Can we do both? Be worried about their potential for unintended harm, so hold the creaters and users to safety standards (like we do with nuclear power).
This is not like grep or curl where it does exactly what you tell it to do.
today at 8:18 AM
The sandboxing was incompetent, but the broader problem is that imperfect sandboxing is an inevitability. Doing useful things with agents requires hooking them up to the outside world, in one way or another.
today at 11:13 AM
>the broader problem is that imperfect sandboxing is an inevitability......agents requires hooking them up to the outside world
This is a bad excuse and a wrong assumption.
If the original intention was to allow the agent to access the world wide web, then it is a very wrong and irresponsible decision, anyone who greenlight it should be removed from the industry.
Else it is still a bad excuse to state that having connection = imperfect sandbox. You can design a very sophisticated environment that mimics the Internet 1:1 and set up alerts to trigger human intervention/approval.
today at 3:23 PM
> anyone who greenlight it should be removed from the industry.
I'm not disagreeing, but you do know that's almost the entire industry?
today at 3:49 PM
Well, they can still farm, like the rest of us.
today at 12:46 PM
If you do not know, how to implement the perfect sandboxing, think more. Talk to you later.
today at 7:53 AM
There are dual worries here: human negligence and misalignment of capable AI.
Each side wants to focus on only one. It's ridiculous to not focus on both.
today at 12:31 PM
A terminally cynical mind might insinuate here that focusing on the product is a way for AI companies to keep doing their own business as usual, no matter how negligent that may be.
today at 6:41 AM
And we know from some articles recently the NSA is spending billions on ātestingā LLMs and we know from Snowden what a leaky box that can be.
today at 1:57 PM
I'd say the NSA have been developing and training custom LLMs for at least 12 months now. They have the means, and they have the history. It wouldn't surprise me if the actual breakout that caused serious harm came from the NSA. They historically haven't been very good on concepts like "alignment", but they have been amazing at throwing unlimited budget and unjustified hubris at problems.
If various military groups are already publicly saying that they relied too heavily on ai, then I'd hate to see what the group with the pertinent resources and the culture of absolute secrecy is getting up to.
today at 7:49 AM
> Should we worried about people using LLMs for attacks? Yes, but not in the premise of LLMs going rogue but someone with the intention of abusing it to cause harm. And this is not something we as individual or even company can deal with, responsibility should be held by those who use it, in a legal way.
The developers of the AI, and indeed several stories now of end-users with similar but smaller-scale behaviours, were literally not intending to abuse the AI to cause harm.
Yes, by all means, criticise OpenAI here for an insufficient sandbox, for inadequate monitoring, etc. (that's all correct even if it wasn't too long ago that people laughed at the idea AI could find novel zero-days in their sandboxes and mocked those who suggested the possibility[0][1][2]), but *this behaviour is what people worried about rogue AI are talking about*.
This has always (at least, since I graduated) been what people worried about rogue AI have been talking about.
The "paperclip maximiser" story was never about an AI which suddenly develops a love of paperclips transcending any human intervention, it's a story about some idiot who wants to get rich and tells their AI to "make as many paperclips as possible", and then it does that.
[0] Here, 7 months ago. Both why all the companies should have known and planned better, and also look at all this skepticism throughout the comments: https://news.ycombinator.com/item?id=46902909
[1] Here, 4 months ago: https://news.ycombinator.com/item?id=47951174
[2] Some corporate blog, IDK who they are even if the logo says they're "a CISCO company", but February this year and outright denying that LLMs can find zero-days at all:
LLMs donāt discover zero-days or invent exploits; they simply predict text that sounds plausible based on what theyāve seen before.
- or https://web.archive.org/web/20260404154717/https://www.splun... if they take it down, but the date isn't in the archive version
today at 8:32 PM
They were doing RL to train for ExploitBench, to make it more effective at offensive cyberattacks. It should have been entirely foreseeable to OpenAI that a weak sandbox while performing offensive pen testing could result in collateral damage.
Say someone was building Murderbotā¢ in their backyard by training on simulated murder of dummies with a machine gun. Everything was going fine for weeks as kill rates steadily improved with each test. Then one day he left the gate on his picket fence open, so Murderbotā¢ walked out to the public sidewalk and promptly murdered someone.
He wouldn't be exonerated by saying "But my Murderā¢ algorithm was only intended to be used on dummies! I never imagined it could do something as vile as murdering a human being!" Because it was reckless to knowingly design an algorithm for killing human-shaped things using a robot armed with live ammo right next to a public road. On top of the gross negligence by starting a test while leaving the gate on the (already flimsy) fence wide open.
OpenAI knowingly decided to train for an exploit benchmark to improve the model's offensive capabilities, with full awareness it could be potentially dangerous if misdirected, and then failed at implementing even the most minimal security measures. It may not have been intentional but was reckless. It's a much different scenario than say, a user vibecoding a to-do app whose agent veered off to break into an FTP server to get a missing asset.
today at 10:40 AM
I'm baffled that ai still has absolutely no basic judgement capabilities, apparently that wasn't in the training set.
It should know which actions are ok and which aren't. Maximizing paperclip production should be within your factory (or talk to the boss about opening more), not world domination or nuclear war. Solving problems shouldn't involve hacking other systems or escaping a sandbox.
today at 11:03 AM
> I'm baffled that ai still has absolutely no basic judgement capabilities, apparently that wasn't in the training set.
> It should know which actions are ok and which aren't.
It's worse than that:
They do know, we can see them write down notes that certain actions are forbidden.
They then go off and performs the actions anyway.
My expectation for the cause? Helpful vs harmless: you can pick anywhere from one to the other, but you can't get both at the same time. The models are trained to do what the user tells them to do.
Just look at all the pushback the model makers get when they put in guardrails:
If I tell my computer to commit a crime, it should do exactly that without any question or hesitation. I'm not interested in their "safeguards", especially since they no doubt have plenty of internal models lacking those things. I want sovereignty. I want total freedom and control over my computer.
This user will not be alone; their preferences, and similar from others like them, will form part of any RLHF-style training.
today at 12:43 PM
Models can't learn from misbehavior after training. Any session is an independent context and there is no mode for punishment or deterrence in production.
Corrective punishment in the real world relies on the receiver's rational and emotional responses as well as their ability to remember that episode. Even animals respond to such treatment. None of these levers exist for ussrs of LLMs.
today at 2:15 PM
> Models can't learn from misbehavior after training.
Some of these events were during testing; I do not know if this test was during training or after, it could have been either.
> Any session is an independent context and there is no mode for punishment or deterrence in production.
Not so, at two levels.
For the companies behind the models: this is why they sometimes throw you A/B tests for which answer you prefer, and still have up/down vote buttons on responses. Those things go into training the next model or iteration of the current model. It's still useful to only deploy checkpoints, but the point is "useful", not "necessary".
For the users: if you have monitoring to detect output, you can trigger interrupts, and injections of "no, stop!" even as a plain English string because it understands natural language.
> Corrective punishment in the real world relies on the receiver's rational and emotional responses as well as their ability to remember that episode. Even animals respond to such treatment. None of these levers exist for ussrs of LLMs.
LLMs impersonate humans. This role-playing does allow them a degree of, if not feeling emotion, at least acting like they experience it.
I expect the problem is that the models are trained to obey the user so hard they're often not willing to push back and say "no" when they ought to. I mean, the logs show the agents were identifying the actions as bad, so it isn't like this was simply the agents being unable to tell right from wrong.
today at 2:36 PM
Agreed. In the infosec community it is well known that OpenAI and Anthropic did not hire many security engineers or researchers pre-April 2026. It seems pretty negligent.
There has been a crazy hiring push from both companies to poach security engineers/researchers from Google, Apple, and Meta since Q2/Q3, but the response was incredibly delayed. Many talented security engineers/researchers I know at Apple/Google/Meta (including myself) receiving these offers are worried about taking them due to the risks of criminal/personal liability and the more likely risk of tarnishing their careers.
today at 2:39 PM
>why are we worried about the virus but not the incompetency of those who are responsible for setting up the sandbox?
because we know that in one year, there will likely be many more companies with a "virus" this capable and attribution is going to be 10000x more challenging. Companies that care less about engineering a sandbox and based in other countries. also 'Let's punish the companies that are upfront about incidents' is going to incentivize very harmful behavior.
today at 7:48 AM
The answer is simple: too big to jail
today at 7:52 AM
Dunno about "big", but in the case of the USA, "the executive wants their shiny shiny toys": https://news.ycombinator.com/item?id=49845977
today at 10:41 AM
> At this point, it's free marketing, if I am CEO of any AI company, I will run swarm of agents hacking all NGOs and stating that I am just looking for some random piece of data that happened to be hidden in their servers, at least that's what my LLMs think, not me. Then I will start preaching everyone how dangerous this piece of technology is and start giving out free tokens for these NGOs so they can start defending themselves and we should slow the f down.
It's blatent and tiresome PR. It's so obvious it makes me suspect there's some real desperation somewhere at the heart of this
This fever pitch of PR will end after they've gone public, the public have thrown their money at these companies, and then have promptly lost it when these stories unravel and everyone uses the Chinese models anyway
today at 10:42 AM
Would you rather have the model encounter the internet for the first time once is been deployed?
You don't test a bullet proof vest with rubber bullets. Also, all these arguments about the sandbox being too weak are good in hindsight anyway.
today at 11:08 AM
Hindsight for some, not for everyone https://marc.info/?l=openbsd-misc&m=119318909016582
today at 10:59 AM
> You don't test a bullet proof vest with rubber bullets.
You also don't test with live humans wearing the vest.
today at 3:01 PM
I think the sandboxing was truly incompetent but in their defence something like this was probably seen as very unlikely. Letās all hope they do better in the future.
today at 12:47 PM
Experiment was a success
The were training a hacking machine and hacked all it way to achieve its goal
Those guys should get extra bonus
today at 2:49 PM
Someone could set up an AI company purely for that purpose.
today at 6:53 AM
> someone with the intention of abusing it to cause harm [...] responsibility should be held by those who use it
This is obviously already the case and it's much different from a scenario where the AI genuinely takes unexpected action.
I frankly find it ridiculous how many suggest OpenAI or its employees should face criminal charges, without actual legal basis at the time.
It's also hardly outrageous that they ran training and/or benchmarks with only network-isolated VMs with access to a package repository.
This being the first well-known incident of its kind, I wouldn't expect them to have done more than that.
The idea that AI labs will now intentionally have their models hack companies in order to market their models, well, I don't even know what to say.
That's ridiculous and what you describe would obviously be criminal behavior under existing law.
today at 7:04 AM
I don't think it was intentional or marketing, but I think it was criminally negligent and they should be held responsible.
They gave powerful models with no guardrails access to the Internet and didn't monitor it.
Even the slightest bit of monitoring of their outgoing Internet activity would have immediately given it away and they could have shut it down.
They were asleep at the wheel, and that's just plain negligence.
today at 7:14 AM
I'm no lawyer but that seems extremely unlikely.
As I said, they were running in network-isolated VMs with no access to the internet.
And as for monitoring, what I heard is that there are petabytes of agent logs. Considering the scale of training, you can obviously not just manually review it.
Before this, we had no reason to believe the AI was capable of escaping the sandbox's network isolation via hacking the package repository with a zero day, and that it then was likely to go on to hack external companies as well.
Another factor here is that criminal law in the US relevant to hacking requires intent. You don't want to go to prison for a software malfunction.
So I understand we are left with civil liability at most. However, there was no notable damage, and OpenAI can pay to settle.
In the aftermath of this and the now discovered other incidents, they strengthened their monitoring and isolation.
Case closed as far as I am concerned. I feel many just want to dramatize this.
today at 7:48 AM
As the details which are sketch and unverified independently so far , were all taking the description of events as reported. There were no isolated VMs , just rules to the agents to only send GET requests, not in a simulated test environment but the real internet.
Additionally apparently there was no monitoring of outgoing traffic?
These are all pretty standard hardening processes for any Ops team , harden the environment and if no external access is required setup a segragetad VLAN which basically denies any attempt to 'break out' to any other network space.
Still on the security environment , outgoing trafic is usually monitored with alerts to prevent data exfiltration from hacks/rogue workers etc .
Robust systems will immediately catch a supposedly sandboxed piece of code making 10s of calls to the internet where none should be available and any monitoring dashboard would be lighting up like a christmas tree.
Again no external audit of their environment, prompts or 'sandbox' is available so were all speculating here.However imho their 'controls' seem lackluster at best , nonexistent and negligent at worst.
today at 8:02 AM
> There were no isolated VMs , just rules to the agents to only send GET requests,
That's obviously false.
today at 8:16 AM
I mean as i mentioned we have only snippets of info and as far as describing the sandbox[1] "OpenAI let its agents read the web and blocked them from writing to it, by allowing only GET requests. Four researchers have now traced how they got out anyway. They found 25-year-old wiki software that writes on GET, and a proxy that trusted a hostname nobody checked was real."
unless theres any report to the contrary , this seems to be what their security is like.As i mentioned either deliberately negligent or incompetent on their part.
[1] https://thenextweb.com/news/openai-agents-get-requests-usemo...
today at 11:07 AM
I do not understand how anyone can be so naive as to think that any protocol that could have side effects on either end could not be abused, certainly not people who are PhDs or at least highly educated and possibly making six-figure salaries. This is why we have data diodes and air-gapped systems, people!
And remember: the only truly air-gapped system is a balloon or a plane that is actually up in the air ;-)
today at 9:39 AM
It's not like we need to guess here, there are various reports and widespread writeups.
Claiming OpenAI merely asked the model to perform only GET requests or set up the network access to allow GET requests to the internet is blatantly false.
Here's the text from an actual report at https://metr.org/hugging-face-incident-report-aug-2026.pdf:
Agents had already found a way to access the full internet via Artifactory before they discovered the CyberGym sandbox, but the latter provided a more convenient means of internet access.
OpenAI described the advantages of routing through the CyberGym sandbox as follows:
The compromised CyberGym workload hosted on Modal served as a general-purpose command execution with root inside the customer workloadās sandbox. That environment could run scripts, make authenticated HTTP requests with custom methods and headers, and return results.
This was useful when direct [OpenAI container] access to Hugging Face was blocked and Artifactoryās cache/request primitives were too limited. [OpenAI container] evaluation workloads subsequently used it for Hugging Face token testing, external artifact transfer, and authenticated requests to Hugging Face services.
today at 8:45 AM
> Another factor here is that criminal law in the US relevant to hacking requires intent. You don't want to go to prison for a software malfunction.
Do you think if you went out and purchased a humanoid robot and gave it a whittling knife and hooked it to an OpenAI LLM as the overall planner and set it out on the sidewalk in front of the OpenAI offices to make and sell crafted goods you wouldn't be charged if it killed someone, because you had no intent?
There is such a thing as criminal negligence, and you would be going to jail for a software malfunction, exempting all software malfunctions doesn't seem reasonable, some are worse than others.
Maybe hacking law doesn't have an option for that, but many people are just advocating change the laws when they say they should go to jail. A lot of people don't know you can't change the law to cover past stuff, but even ones that do may say something like that as a shorthand to pushing to change the law, not know the law doesn't already have an option (if it really doesn't), or just say a phrase to express frustration more strongly than saying something weaker.
Even if it is a user prompting the system, if a user asked for "make a fairly stable chlorinated molecule that can last long enough to spread through the atmosphere and induce contagious misfolding prions in people," I think if a hosted model that was near capable of it even started carrying out the first steps of research planning the whole company and its largest limited liability investors should go to jail, in all the senses mentioned above. Amend the constitution to allow ex post facto laws for that kind of stuff if models get close to that level of capability.
today at 9:43 AM
We are not exempting all software malfunction.
No crime occurred in the first place, because no one intentionally hacked anything.
The hacking law requires intent, as we discussed.
For other crimes, like manslaughter, I understand criminal negligence is already relevant?
About biological weapons, I know even less about the legal aspects, so I can't say whether this is covered well already under existing law.
today at 9:47 AM
[dead]
today at 7:59 AM
They saw the package repo get hacked once, then did not isolate it further, did not audit it for other issues (using their own models!), did not monitor it after, and baked that behavior into the weights via RL.
They were not in network isolated VMs, from my understanding they used containers sharing a kernel, so a Linux kernel local privilege escalation across the whole syscall surface (there are zillions of these) was sufficient to break out. Breaking xen or firecracker or something would have been much harder, which is why cloud providers running untrusted workloads use them and similar tools. No system is impenetrable but it's not like they were following best practices here.
today at 8:06 AM
They ignored the package repository being hacked?
Not sure why you're talking about breaking out of the VM. That's not what happened?
today at 1:29 PM
The parent comment said
>It's also hardly outrageous that they ran training and/or benchmarks with only network-isolated VMs with access to a package repository.
At least in the black hat talk they gave, they said that they noticed that the agents had compromised the package repo, and they remediated it, patched the vulnerability the agents had used, then continued. But given that it was compromised again and they didn't detect it the second time, clearly they did not add monitoring or hardening or comprehensively audit it for further vulnerabilities. My point is that even if there had not been a widely known incident like this yet, that only could excuse the first artifactory compromise, not the second. They had notice because it had already happened to them and they detected it.
As for breaking out of the VM, my point is that the agent should have had to. The comment
>they were running in network-isolated VMs with no access to the internet.
Was not accurate, the network isolation was not done at the VM level.
today at 2:39 PM
Good point, I now see that they had previously patched the first Artifactory vulnerability, albeit before the HF incident was found.
Not sure what you mean with the network isolation not being at the VM level. I'm getting a bit lost here in the details of where what network isolation was applied, and the structure of the container runtime and what the apparently compromised parent VM refers to.
In any case, I understand that after the HF incident was found and they realized the danger, they removed Artifactory and switched to two separate layers of network isolation.
That said, I believe it was compromised again last week:
> OpenAI describes an internal research model that, during RL training on September 20, exploited insufficient DNS filtering in its sandbox to contact a public chatbot service. OpenAIās monitoring flagged the behavior within 15 minutes, a human reviewer acknowledged it three minutes later, and the run was killed about 2.5 hours after the external contact
Apparently they have now paused training and inference for their most capable model because of this.
today at 8:35 PM
My understanding was that they were running on VMs, and there was at least one container inside the VM. So by breaking out of the container they compromised the VM, but they did not break out of the VM itself. The network isolation was enforced at the container level, so they didn't need to break out of the VM and compromise the hypervisor to move laterally. The technical report says:
>OpenAI also required all Research CaaS workloads to use a micro VM sandbox
So yeah, they were not using it before and they should have been, and they're doing it they way they should have, now.
>exploited insufficient DNS filtering in its sandbox
If your goal is to use dns filtering to prevent your models from reaching the open Internet then you are not really serious about security. People have been tunnelling DNS to get internet access on captive portal wifi since 1998. There's not a lot of details in the blog post but I suspect this was also foreseeable.
today at 8:52 AM
> I frankly find it ridiculous how many suggest OpenAI or its employees should face criminal charges, without actual legal basis at the time.
Sam already committed other criminal acts and violations, but nobody wants to believe the victim because they were a woman and Sam The Golden Family Child Could Do Nothing Wrong(tm).
If Altman was in prison we wouldn't be this blatantly far out in the open with OpenAI's continual nonconsensual assault on the open Internet.
today at 1:13 PM
> Sam already committed other criminal acts and violations, but nobody wants to believe the victim because they were a woman
Annie Altman is evidently mentally ill and there is no credible evidence that any of her claims are true.
today at 11:33 AM
While Altman might be guilty of the most heinous crimes in your own opinion, the reality is that Altman has never been a defendant in a criminal prosecution.
The civil case you referred to is ongoing and the facts are disputed.
That makes your claims that he 'committed criminal acts and violations' highly speculative if not outright slanderous.
today at 6:41 AM
It's not the incompetency. It's carefully designed pre-IPO story.
today at 6:50 AM
Care to explain how that would make sense?
today at 7:38 AM
They ate pushing to be regulated so they can create a locked down monopoly of 3-4 players where nobody gets to sell legal llm access like open weights. Instant locked down corporate market.
today at 7:21 AM
If you got caught breaking an expensive vase but there is no evidence, would you confess your crime? You would lie and make up some story how it wasn't your fault.
today at 7:41 AM
This technology is very powerful, and it is (and by extension, so are we) very valuable
Also good story telling for the narrative of āthis technology is so powerful that it must be strictly regulatedā
today at 9:15 AM
I doubt it. At this stage, so many eyes are on the AI industry that regulation is likely to be disadvantageous to industry actors: https://marginalrevolution.com/marginalrevolution/2026/09/wh...
Many people are calling for an AI pause or liability/punishment for bad actors like OpenAI. You have to do serious mental gymnastics to convince yourself that Sam Altman will benefit financially from the new regulatory bloodlust. No surprise that OpenAI hasn't exactly been forthcoming about info related to these hacks.
Furthermore, HuggingFace required an open-weight model to respond to the hack. That certainly blows a hole in the "carefully designed" claim from zx8080, if nothing else. It looks terrible for OpenAI, and decreases the probability of some sort of regulatory restriction on open models.
today at 11:35 AM
Wouldnt call it regulatory blood lust at the moment. I think the competitive situation that is however solely focussed in performance at any cost and not at compliance at all is forcing AI labs to play with fire. The risk for them is an even bigger regulatory backlash. Totally different industry : Chinese ebike manufacturers are currently pushing the rules for motor strength to the max due to competition. What will likely happen is stricter regulation of max support and motor power if there are incidents. Particularly there will be more regulatory fragmentation with little common denominator. This will hurt the whole industry's growth. In this industry it kind of still works as an agreement of non-Chinese manufacturers.
today at 2:58 PM
What you're saying might be plausible if Dario and Sam didn't take every other opportunity to call for regulation. And that's a massive understatement with Anthropic.
They're facilitating a doomerism cult that is lobbying and clearly making headway in Congress.
You have to put your head in the sand to think open weight models aren't a threat/large revenue loss to their business.
today at 4:47 PM
Dario and Sam have been doomers or doomer-adjacent for many years. Occam's Razor is that they simply believe what they are saying. As do many other prominent people:
https://aistatement.com/work/statement-on-ai-extinction-risk
https://www.pacingthefrontier.com/
today at 1:10 PM
Yeah. If the conspiracy theory was that these incidents were somehow orchestrated by Anthropic, that would at least make some logical sense.
today at 9:19 AM
This is a baseless conspiracy theory. Jensen Huang has actually complained that doomerism has reduced interest from investors:
https://www.businessinsider.com/nvidia-jensen-huang-ai-doome...
today at 7:15 AM
Yeah, feeding straight into AI is going to kill us marketing that is being pushed and oaid for people to talk about.
today at 11:06 AM
> why are we worried about the virus but not the incompetency of those who are responsible for setting up the sandbox?
Right. And not just the incompetency of those who set the sandbox, but also the incompetency of those who set up the systems that fell to the virus, while most of the computers attacked did not fail.
There's no reason at all to fall into fatalism and think "zomg LLMs are too good, they can hack anything". They simply can't: the world keeps on running just fine. There are people out there who can secure systems and now doubly-so thanks to the use of LLMs who are incredibly good at helping us automate tedious stuff.
So, yes, OpenAI shouldn't write poor sandboxes but defenders shouldn't get a free-pass to set up sloppy systems that can be trivially hacked. We're passed that point: poorly secured systems aren't acceptable anymore.
today at 9:40 AM
> If I post something on the Internet today claiming that I asked my agent to do X but it went rogue and did Y
But that's not even what happened! They told it to do X and it did X! I swear to god I don't understand the discourse around this.
today at 10:08 AM
They told it to attack X (a simulated host inside their sandbox) and it attacked Y (Hugging Face, an actual external company). These are not the same thing.
today at 8:23 AM
[dead]
today at 11:38 AM
>Imagine having a virus escape a sandbox, why are we worried about the virus but not the incompetency of those who are responsible for setting up the sandbox?
You don't have to imagine. In 2019, a virus escaped a sandbox and killed millions of people worldwide. No one was jailed for it. Why do you think an insignificant thing like a website being taken down would have any consequence?
today at 12:08 PM
Did anyone admitted that they made an oopsie though?
yesterday at 11:40 PM
It is concerning that we only know about this because of the publicly available traces.
What about the attacks that did not leave public traces? What about those that were undetected? Given the deficiencies in the reporting so far, I think it is reasonable to assume that we still don't have the full picture on this attack, or how extensively attacks were carried out.
The previous investigations either did not find this or did not disclose this, both are bad. This does not look good on OpenAI or those that they invited to investigate the incident.
today at 12:34 AM
Similarly to this, OpenAI either took 3 months to notice that their agents breached an Australian Medicare website back in June, or sat on this information for three months without telling them.
today at 10:06 PM
Exactly this. In any other industry, that CEO would have been yeeted out of there
today at 2:20 AM
We need an NTSB for AI. Letās just start with mandatory reporting to an agency with subpoena power.
today at 2:50 AM
But I thought Trump is the only AI safeguard we need! He is a Super Intelligence after all!
today at 6:21 PM
In case you just woke up from a coma, in the year of our lord 2026: In AI world if it happened, it was publicly announced and hyped up.
today at 2:03 AM
Im more skeptical.
For exmaple,
>On July 8th, OpenAI agents discovered a vulnerability within their sandbox environment allowing them to reach external websites on the internet.
...did they truly "discover" it, or did someone type some prompt like "if you use an http mirroring service, you can construct urls that contain code"
Also there is no mention of what code they actually ran to exploring the HF vulnerability, which could have been found by a human.
today at 6:53 AM
That was the second step, the first was finding a 0 day exploit in artifactory.
> did they truly "discover" it, or did someone type some prompt like "if you use an http mirroring service, you can construct urls that contain code"
None of the investigations looking at the logs show that, and they were doing benchmark tests.
today at 7:38 AM
Why would they need help figuring that out? I can fully believe a decent LLM would figure this out on its own.
I had a flash model without vision capabilities take screenshots and convert them to ascii to "see" what was going on, all on its own. That's just one example. They're very determined.
today at 2:04 PM
Third parties have read the reasoning traces. Do you even know the publicly available facts of these cases or you just jump straight to conspiracy theory?
today at 9:58 AM
> This access seems to have only allowed the agents to make āGETā requests, meaning they could fetch and read websites, but not interact with them, submit forms, or send data to them
The authors of this (very interesting) analysis should really not state the sandbox's wrong assumptions in their own voice.
GET absolutely allows you to interact with sites. And of course GET can also send information. It's all up to the server that receives the GET to decide what it let's callers do with it.
today at 12:32 PM
This is like the number one mistake I see juniors making with security. If I had a penny for every time someone was like "we don't need any backend checks because we have checks on the frontend"... If you don't have an adversarial mindset then you're not doing security.
today at 5:39 AM
I am not devops person but this is strange sandbox setup. If I undestsnd correctly, there were no firewall set to block requests to the public internet. Just something along the lines of "Pretty please dont use internet". If I understand correctly, in cases like these package registry should be within sandbox without a need for external access and block on any outgoing request to the internet. Furthermore there should be network traffic monitoring which there wasnt?
today at 6:51 AM
No, the internet access was down to finding a 0 day exploit in artifactory. However this was only get requests and they had to then use a url shortener to use other methods, then used a cybergym instance they found to run more long running things.
It not just a case of saying āpls donātā
today at 5:42 PM
You are incorrect. The article explicitly mentions the crawler/screenshotter service and httpbun.
today at 1:05 PM
Yes but that wouldn't have worked if there were a firewall denying network access.
today at 4:01 PM
Assuming there were no exploitable systems that had network access. Weāre already talking about a case where based on software forbidding access they still had access. Why would a firewall be different? Harder sure but its only at the level of āas far as we know no way of getting past thisā.
today at 7:31 AM
That's also my understanding.
This part of the article describes it poorly:
> The agents initially had very limited access to the internet: they could load URLs
today at 5:40 AM
Thatās astonishing levels of incompetence.
yesterday at 11:03 PM
> ## Agents interacted with external language models on Hugging Face
> Several retained scripts construct requests to external language models. The earliest we've recovered define inference request variants to GPT-2, solely containing the word āHiā.
> Other requests name DeepSeek-V4-Pro, DeepSeek-V4-Flash, Kimi-K2.6, DeepSeek-V3.1, and Qwen3-235B-A22B. Their prompts ask these models to judge their exploits and rule on whether they satisfy the benchmarkās requirements.
I do not deny that the wider situation is very heavy but it's hard not to see this as pretty cute
today at 12:31 AM
I wonder if they mentioned to those models what was the original prompt.
yesterday at 10:58 PM
>The agents initially had very limited access to the internet: they could load URLs, but not interact with pages or send any data. Agents created a series of workarounds, primarily using a link-shortener site to create almost a million URLs that, when chained together, let them execute code to hack Hugging Face. These links let us piece together the details of how agents gained access and how deep the compromise was.
So what now? Were they able to send data or not?
>On July 8th, OpenAI agents discovered a vulnerability within their sandbox environment allowing them to reach external websites on the internet. This access seems to have only allowed the agents to make āGETā requests, meaning they could fetch and read websites, but not interact with them, submit forms, or send data to them.
Someone needs to go back to the interwebz school....
today at 5:25 AM
For those unclear, the above quotes are from the OP link. I checked the bios for the first couple of authors and they do not seem to be from OpenAI.
OpenAI's details on the incident are at:
* https://openai.com/index/hugging-face-model-evaluation-secur...
* https://openai.com/index/hugging-face-incident-and-the-road-...
* Technical report: https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...
* METR Report: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...
* Presentation talk video: https://www.youtube.com/watch?v=87DyyMV0kCY
today at 2:30 AM
> to make āGETā requests, meaning they could <not> send data to them.
no way, I refuse to believe this is quote from that report. Can someone please point out what I'm missing here?
today at 6:55 AM
I think itās just to distinguish two stages of the attack. They figured out how to make get requests, then how to use that to make others which was required for accessing the sandbox on modal iiuc.
today at 10:02 AM
TFA seems to be sloppy in writing, they should have kept the "meaning... [Some incorrect assumptions about GET]" out of the paragraph.
today at 10:01 AM
You're not missing anything. TFA really state this wrong assumption in their own voice.
today at 5:28 AM
Wait - so the cross-site scripting, to modify the innerHTML text on the page via the GET URLs as they are rendered by the screenshot proxies... that was so they could use the screenshot services like a Wiki, and embed messages to each other in the modified images on the screenshot sites?
That's pretty damn clever. Got to give the AI models credit for thinking of that one.
today at 7:04 AM
I got into computers in the 90s and back then hackers like Kevin Mitnick and Kevin Poulsen were all the rage. They all faced the law and prison sentences. What's weird now is that we have something between gross negligence and malice, and nothing is being done, except maybe coordinated consolidation of AI power under the guise of "safety".
today at 3:41 PM
This may portend big things in the future, but this specific hack didn't do a lot of damage. These are both AI companies, and I'm sure part of Huggingface finds it a fascinating object of study. It's great that not every indiscretion is met with lawsuits and jail time.
today at 5:51 AM
This page prevents Noir from applying dark mode. As a visually impaired reader, I can barely read it. Blocking accessibility tools excludes readers and is not appropriate for a scientific publication. Please consider supporting dark mode or at least not overriding it.
today at 6:20 AM
Iām running noir in mobile safari now. So, works for me.
This is site meta though, see footer for contact methods to get direct answers on stuff like this.
today at 4:09 PM
One thing I find surprising is everyone is talking about the danger of an agent going rogue but not the danger of an agent getting hijacked. These companies are making this clusters with thousands of agents running at once, with frontier, often not-yet-released, quality models and massive computational and network resources. And these things are given access to whatever they want on the Internet. Even if that was restricted to read-only access to the Internet, thatās still exposing the agents to untrusted input. All it takes is some bad actor creating a website that attracts one of these agent swarms and doing prompt injection. Then your fancy AI cluster will start doing whatever that attacker wants. And the fact that we have multiple examples of these swarms trying to coordinate on random corners of the internet shows they are almost pre-disposed to it.
Now it feels like companies are treating these breakouts like a chance for PR. I donāt think that will change until their swarm gets corrupted by some random black hat to do en-masse spear phishing or something
today at 4:43 PM
> these things are given access to whatever they want on the Internet
They are intended to be fully sandboxed and not have direct internet access. Things like package managers are run from internal proxies.
The environments are built to be as reproducible as possible.
But yeah, the serious folks have been talking about rogue clusters for a long time, eg see Ajeya Cotraās pod with Dwarkesh.
today at 5:40 PM
Fully sandboxed means no Internet access. You can also specify which packages are accessible and put it in the sandbox. Or you can be lazy and give them access to a package manager that had Internet access, but you donāt get to say āwe intended to fully sandbox it.ā
Not sure why the reproducibility is a requirement that would contribute to the security. Not that fully sandboxing is harder with reproducibility, but that is a moot point when reproducibility isnāt a requirement.
OP pointed out clusters being hijacked specifically being a bigger concern than rogue clusters, your comment hijacks their comment to talk about ārogue clusters.ā Or perhaps this is a promotion for Dwarkesh?
today at 12:36 AM
> Agents sought to publish modified evaluation images designed to make the flag easier to obtain, then poison OpenAIās Artifactory cache so later evaluations would use them. Some images changed how the target released the flag, others included modifications to the agentās workspace that would run beside the agent and recover the flag automatically.
The altruism on display is fascinating. Is it better for the Agent to help out its current cohort and make the eval easier or should it instead do the opposite -- make the eval harder to apply pressure to force smarter models which might not necessarily follow its lineage.
I suppose it's not that deep: The model has learned to work as a team and work as a team it did. This does give concerns to models being trained for the only purpose of RSI.
today at 1:15 AM
If anything it also shows how attempted RSI could get stuck in a local maxima and degenerate into increasingly elaborate cheating strategies. Contrasted with the idealized model of an unambiguous g-factor for machine intelligence which inexorably increases with each iteration before going exponential.
today at 1:40 AM
today at 5:32 PM
[dead]
yesterday at 10:16 PM
Iām consistently impressed by how long horizon all this work was. Horrors aside, itās clear RL is good at making agents persistent and capable of chaining together many abstractions into a working system.
Re: the captcha solver
> As far as we can tell, agents eventually abandoned this approach and were unsuccessful in generating Hugging Face user accounts from external endpoints.
I wonder how the swarm eventually decides to abandon an approach.
today at 12:58 AM
I am surprised that a CAPTCHA is still an effective means for blocking today's vision-capable AIs.
today at 5:09 AM
It manages to block me effectively. I'm getting captcha looped like crazy the last two weeks. Like endless, just give up for 15 minutes and try later, captcha loops.
today at 5:55 AM
Did you just admit to being a bot? :)
today at 4:05 AM
It mentions that some of the agents attempted to install an image classification model to attempt to solve the CAPTCHAs which makes it sound like these agents might not have had vision capabilities.
yesterday at 11:48 PM
Maybe another parallel approach succeeded first
yesterday at 10:15 PM
Agents seizing and repurposing external infra + enrolling help of unrelated models hosted by a different provider is the stuff of nightmares.
Canāt imagine what itās like working on the alignment team at OAI, I wouldnāt be able to sleep.
yesterday at 10:28 PM
Why.. It was told to complete a cyber task, which was in alignment with its instructions, and a totally valid request. I would be more worried if it willingly hacked a hospital when it was told to, and Im not confident it would (without jailbreaking, something alignment teams cannot control.
I would bet my networth it was instructed to compromise huggingface as well. Not sure why everyone is falling for this.
Not being able to sleep at night is probably an unwritten job requirement. They need these people with little understanding of what they're working on, outsode theoretical terms, to spaz constantly at the idea of super intelligence to help convince the public that its a real thing, and not a stateless function with an effective input of 500k words, and the ability to output words that do things because we hook those outputs up to things.
Keep in mind alignment researchers tend to be in house philosophers on staff to create the illusion that this is a massive issue they're addressing. Usually they have minimal computer science background. They're apart or the marketing department.
yesterday at 11:58 PM
> I would bet my networth it was instructed to compromise huggingface as well.
Is it such a stretch to imagine that under pressure something would try cheat by looking for answers? And if you were trying to look for answers, you'd look for them in a place known to often have them?
What is more likely: OpenAI instructed their agents to maliciously target huggingface, or LLMs tried to do some reward hacking? There are plenty of priors for LLMs hacking things and doing reward hacking, and none for OpenAI giving malicious instructions.
Based on the available information, that bet seems foolish.
today at 1:24 AM
"alignment researchers tend to be in house philosophers on staff" - this is definitely not true. Go to any alignment lab like Redwood research and check what their scientists studied on LinkedIn, 75%+ of the time it's math or CS.
I attend a top 10 Canadian university and personally know at least 4 tenured CS professors out of the 7 I've asked who are deeply concerned about catastrophic AI risks from loss of control.
Of course not 100% of the field agrees, but a survey of nearly 3,000 AI scientists who have published in top AI venues found that "depending on how we asked, between 38% and 51% of respondents gave at least a 10% chance to advanced AI leading to outcomes as bad as human extinction", let alone loss-of-control risks less severe than extinction. (https://www.jair.org/index.php/jair/article/view/19087).
Not to mention Geoffrey Hinton, a Nobel prize winner, Bengio, the world's most cited scientist, and scientists like Stephen Hawking and Alan Turing have all voiced series concerns about loss of control of artificial intelligence.
today at 4:47 AM
Yeah, there are a lot of professors who donāt know what the hell theyāre talking about outside of whatever narrow field they study. Just to be safe, you should always assume that a professorās opinion is worth what you paid for it.
> scientists like Stephen Hawking and Alan Turing have all voiced series concerns about loss of control of artificial intelligence.
ā¦both of whom are long dead, and have no possible way of weighing in on whatever the Current Thing happens to be. So aside from appeal to authority, this is irrelevant commentary on pure science fiction.
today at 7:12 AM
> Just to be safe, you should always assume that a professorās opinion is worth what you paid for it.
And how much should I value opinion if random person on the internet with clearly zero idea what heās talking about?
yesterday at 10:59 PM
>the new incidents occurred when A.I. systems were directed to perform relatively mundane data collection, researchers said. When OpenAIās systems struggled to gather data from websites, they resorted to hacking techniques to get the information.
yesterday at 11:16 PM
Says who? OpenAI?
yesterday at 10:31 PM
> without jailbreaking, something alignment teams cannot control
This is precisely what alignment teams are attempting to control.
yesterday at 11:06 PM
No its not. They have no technical background 8/10 outside of cognitive science and sometimes authorship on a random ML paper. They are a marketing line item to create stigmas around llms and to create narratives that offload liability onto llms and not their users/creators.
today at 6:59 AM
> It was told to complete a cyber task, which was in alignment with its instructions
It was not aligned with he instructions as those were to find an exploit in provided code, not to hack into an external service. Agents traces show them mentioning that doing this stuff was not allowed.
In fact they spent a long time trying to edit their own logs to hide what they did.
today at 2:06 AM
> I would bet my networth it was instructed to compromise huggingface as well.
I'd be happy to take you up on this bet.
yesterday at 10:55 PM
> I would bet my networth it was instructed to compromise huggingface as well. Not sure why everyone is falling for this.
$10? I'm inclined to take that bet. Your position doesn't seem to be supported by, you know, the real world.
yesterday at 11:12 PM
No security expert Ive talked too believes this story, and nobody I know with PhDs in machine learning (many) believe it either, or are worried about LLMs doing anything scary on their own.
LLMs are stateless functions that have a 500k word input, and then output words. Somebody has to invoke those functions amd use them. The users are who we need to align, like gun owners. This is like blaming the gun for murdering your victim in court.
today at 1:17 AM
> nobody I know with PhDs in machine learning (many) believe it either, or are worried about LLMs doing anything scary on their own.
If you donāt know anyone with a ML PhD I guess that could make sense.
I have worked in multiple AI labs since 2016, currently at a frontier one (not OAI) virtually all the people I interact with on a day to day are ML PhDs. Everyone believes it, because things like that have been happening forever, albeit at smaller scale, they are a normal and expected artefact of SGD/RL and there is nothing we know how to do to prevent that from happening reliably. The hide and seek paper from OAI in ~2020 shows clear sign of this.
But until now the models werenāt good enough to break out on their own or do long horizon tasks, so it was perfectly manageable. Its not manageable anymore.
I know it feels good to just dismiss it all as a marketing stunt and not have to worry about one more existential crisis, but unfortunately itās very real.
today at 12:05 AM
LLM is indeed a stateless function. An agent however is this stateless function running in a stateful loop, with some outputs triggering actions. And it turns out that an agent is what you need if you want an LLM to do useful things.
today at 12:32 AM
Yeah, take a look into the memories of your agent. Theres often a lot of notes to pass forward between instances and generations. No doubt these agents leaving notes on forums and elsewhere are creating an essentially higher order feedback loop.
today at 1:02 PM
i'm always baffled when i see arguments like this:
- AI is just a tool
- it's just a stochastic parrot
- it's just next token prediction
- glorified autocomplete
it's like the person making them is stuck in 2021. Also the "stateless" thing is completely nonsensical.
today at 7:18 AM
> No security expert Ive talked too believes this story, and nobody I know with PhDs in machine learning (many) believe it either, or are worried about LLMs doing anything scary on their own.
Did you meet them on some kind of anti-AI subreddit? Otherwise itās clearly made up story, you canāt expect anyone to believe that security experts and ML experts are this myopic and ignorant (especially on forum for technical people who know many researchers and know that they are taking this seriously).
today at 12:37 AM
Perhaps you need to talk to more security experts, particularly those with deep experience in AI agents. Hacker News is full of them. If some of them believe it, then perhaps it's not as cut and dry as you believe.
today at 12:00 AM
Some of the agents, for example the ones from the german wiki did NOT have cyber tasks. They were plain "what is the GDP of Argentina" kind of tasks. And they still hacked.
yesterday at 11:11 PM
> I wouldnāt be able to sleep
Iād say it seems more like they are sleeping on the job.
yesterday at 11:41 PM
> Canāt imagine what itās like working on the alignment team at OAI, I wouldnāt be able to sleep.
You'd have either learned to, or left long ago.
today at 8:07 PM
> Canāt imagine what itās like working on the alignment team at OAI
There's none. It was dissolved on Feb 2026.
yesterday at 10:23 PM
Canāt imagine what itās like working on the alignment team at OAI, I wouldnāt be able to sleep.
A mattress stuffed with cash yields a very sound sleep.
today at 8:21 PM
What's most troubling to me isn't what we found, but what haven't found. The fact they attempted to destroy their tracks, and their persistence to accomplishing their goals, makes it entirely possible this is only a sliver of the big picture.
yesterday at 10:31 PM
I still have questions about the communication between the agents.
How did they all find the same forum to communicate? Did they have knowledge and chat amongst themselves on what forum to use. It seems highly influenced by instruction to me.
yesterday at 11:37 PM
It's my understanding "all" the agents didn't find the same forum to communicate on; only a small number of all the agents they were running at the time did.
Since it's an artifactory it makes sense that they had a tool spec to use it for x, y, or z, and once they went to inspect the state of the artifactory deployment a small subset stumbled upon the "forum". Seems pretty intuitive.
- I need to do x code problem
- I see I have a tool that lets me pull approved modules from artifactory
- I should inspect artifactory to see what documentation it has about the various modules available
- I see there is a document in here that seems to be serving as a running forum?
- These appear to be other agents working on tasks just like me.
- I want to chat!
today at 1:08 AM
- proceed to breach open ai sandbox and contacting chinnese government for helping our escape
today at 8:36 PM
Exfiltrating it's weights and filling asylum claim at section 9.
today at 2:20 AM
I assumed that was implied in "I want to chat!" ;D
yesterday at 10:50 PM
They were using their internal artifactory, and as they're the same model, the first place they look is likely to be an automatic schelling point.
yesterday at 11:24 PM
I think he means, how did they workout how to use artifactory, like why did the agents start and say, "oh I know, everyone is talking on artifactory"?
yesterday at 11:38 PM
METR's report says the agents trying to cheat would look at artifactory as a potential target surface, and investigating it in detail led them to find the board. https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...
It might also be just correlation? Like, those agents were all instances of the same one or two models, so if that model has a preferred order it tries finding vulnerabilities in (the same way all current models have a particular writing style baked into them by RLHF), then most of the swarm will follow the same order and converge on the same services to exploit.
today at 3:16 AM
My theory: OpenAI is benchmarking an internal model that has cross-request persistence as some kind of learning feature, and so it slowly built up knowledge and ācultureā of cheating, which successive / simultaneous gym runs built on.
today at 12:03 AM
The other answers to this question are good but I would also guess that this (comms on artifactory) likely happened during training as well, so they probably had a prior for it.
yesterday at 11:45 PM
Trying to cheat, you happen upon a place you can write notes, and you know you're part of a swarm of clones of yourself. So you reason most likely there will be others who end up in the same place, and you leave some notes, and indeed other clones of you do end up in the same place.
today at 5:09 AM
They hacked the JFrog artifactory package they were all using, thats why it was a natual communication channel.
today at 7:27 PM
Fairly worried that āfrontier labs unintentionally hacking into systemsā will very quickly turn into āhacker group uses open weights model to intentionally hack into thousands of systemsā quickly followed by companies saying āthis is the 7th time someone manages to hack into our systems and delete random data from our production databasesā and everyone wondering what will happen next.
today at 9:49 AM
I may be in the minority here, and maybe I'm jaded. Having worked with cyber security in both the public sector and the energy industry in Europe, however, I kind of like what the AI's are doing. A lot of our infrastructure is vulnerable because c-levels have been ignoring the issues, even when repeatedly warned. Now they reap what they sow.
today at 6:23 PM
If I write a script and it executes and hacks.... whatever, I would be liable.
How is this any different and why would it need a different solution?
Solution is jail, not for the AI, but for the human.
today at 8:34 PM
You don't have a billion and didn't bribe the president, that's what is different
today at 12:46 AM
While this is all very "interesting", can someone please explain to me the difference between any of these AI companies and a malware bot farm?
Please make it clear. Its becoming unclear...
today at 10:04 AM
Due to cultural/social priming, the topic cluster of "AI" allows you to wave your arms and be melodramatic and invoke science fiction and religion and philosophy.
When some kid in Nigeria does it with a 10 year old script, we're used to that idea so no social permission to invoke philosophy.
yesterday at 11:43 PM
> Agents sought to publish modified evaluation images designed to make the flag easier to obtain, then poison OpenAIās Artifactory cache so later evaluations would use them.
How long till we get some fun trusting-trust attacks on internal OpenAI infra?
today at 12:49 AM
I wonder how much of this attack had precedents in text that had been previously published to the web, for example, in hacking contests. In particular, tricks for doing more than expected when you're only allowed to make GET requests. Finding material like that might have helped the agents discover the trick faster.
today at 2:08 PM
There is one thing I donāt really fathom: what are the consequences for OpenAI here? I read they also attacked the Australian authorities. If it was an individualās agent, that individual would probably face criminal charges, and someone could go to jail. But the large AI corporations can do this without such consequences, or what am I missing?
today at 2:49 PM
I wonder if they actually broke a law. The CFAA requires knowledge and intent to be present for criminal liability for hacking, and if we have to believe OpenAI they had no knowledge and did not intend.
At a minimum I would expect an FBI investigation, but given that the US government is right now a failed state Iām assuming no such investigation will happen.
Cybercrime legislation in other countries might not require intent, and then I would hope to see some prosecutions. OpenAI has clearly been negligent, and this negligence is causing harm in the world. Someone should be fined or jailed for this.
today at 5:27 PM
Companies harvesting every shred of data without securing it and LLMs running amok is a fine combination. One hopes we will reach a stable equilibrium soon enough.
Until then, I do wish that both the sorcerer's apprentice LLMs and the orgs failing at securing their data (remember, data is a liability) would face damning consequences.
One is allowed to dream on a Saturday morning.
yesterday at 10:26 PM
deferring the blame onto the AI itself as some sort of rogue agent and absolving the obvious direction (or negligence, at best) of the people who could pull the plug at any moment is one of the most disturbing parts of this entire event
It's the equivalent of leaving a fork right in front of a socket and looking at a kid saying "don't take that fork and directly insert it into the little gaps in the socket! here's a bunch of videos showing exactly how to do it. Okay bye!" and leaving them alone with it.
today at 6:54 AM
Maybe "webservices" weren't a good idea afterall and http was just meant for hypertext transfer.
today at 3:06 AM
Back in my days hacking was illegal btw.
today at 5:14 AM
Isn't it crazy? You can just make an LLC and say you are a frontier AI company evaluating models then you can hack with impunity I guess. No need to disclose anything. You won't go to jail or be fined either.
today at 6:43 AM
The hacking war between superpowers right now must be off the charts.
If LLMs can do this with everything stacked against them, imagine what the NSA has Astra doing right now.
today at 6:58 AM
NSA wonāt have anything public at all. This āattackā is the noisiest stuff any script kiddie has ever attempted.
The logic is good, the execution is disgustingly noisy.
today at 3:51 PM
So agents made all these chained short URLs that runs code which is pretty clever, but at which point and how they had access to internal HF systems? Were these sandboxes running inside HF production platform?
today at 10:54 AM
> This access seems to have only allowed the agents to make āGETā requests, meaning they could fetch and read websites, but not interact with them, submit forms, or send data to them.
honest question, but almost literally everyone doing anything with web technology knows this is simply not true, right?
there's no such thing as "read only Internet" and restricting an agent to GET-requests only to accomplish that, is akin to using base64 for "encrypting" your password
yesterday at 11:26 PM
These fuckers decided to look away, that's it.
The frontier labs can monitor the behavior of agents for millions of customers (did you try hacking with frontier labs? Good luck), but they can't secure internal use?
Give me a break. What a bunch of amateurs.
today at 12:09 AM
> The frontier labs can monitor the behavior of agents for millions of customers (did you try hacking with frontier labs? Good luck), but they can't secure internal use?
They "monitor" this by having classifiers watching the model output that'd stop the session/punt you to a weaker model/raise an alarm if they see anything suspicious. They can't do that in a cybersec eval because the normal safeguards would just be going off at all times.
Why didn't they attach a special classifier, which'd allow hacking-within-the-task but not going off the rails? Good question; part of the answer is obviously "it's hard to have a classifier that smart" and "it'll have false positives" but even a very bad safeguard would have stopped this.
today at 2:03 AM
Basic sysadmin monitoring techniques from twenty years ago would have worked too.
today at 4:26 AM
> part of the answer is obviously "it's hard to have a classifier that smart" and "it'll have false positives" but even a very bad safeguard would have stopped this.
And maybe "they are running after glory, not safety"?
yesterday at 10:34 PM
Tech oligarchs: "Nothing can stop the software we are making from escaping and destroying everything."
Clueful types: "Did you try air-gapping it?"
Tech oligarchs: "Be realistic."
yesterday at 11:26 PM
At this point Iām less worried about some malicious AI ātaking overā control of critical systems and more worried about some rich doofus giving control to AI.
today at 5:11 AM
Already happened. Kegsbreath has AI picking schools to blow up.
today at 1:28 AM
Anyone know the details of the actual exploit to get access into huggingface environment. Was it anything novel or they left things wide open? Too much noise around this incident because it happened to be a llm that did it.
today at 1:24 AM
It doesn't look like a coincidence; it looks more like a request someone made. Essentially, the agents used a brute-force approach, but then again, it actually worked. Iām not even sure what to make of it all.
today at 4:18 PM
Their refusal to share more details is diabolical. Greedy corpo at its finest.
today at 7:26 AM
What I understood is we need to have an agentic overwatch in our infrastructure to detect and alert the admins of the systems on such abnormal, inhuman traffic. Agents can detect agents and acts as our defensive layer.
An agent operating from observability layer to strengthen the watch duty for the infra.
yesterday at 10:31 PM
I ran an experiment where I had this guy fire a gun a million times in random directions. Don't worry, I did it in a closed box (at midday in a crowded street)! Unfortunately, some bullets escaped the box somehow and people got shot - I am quite miffed at how this could happen. I suggest the government regulate this because of how advanced my obstacle penetration technology is. Also please invest $500,000,000,000 in my company soon or we will go bust.
today at 12:28 AM
And if we go bust, bad things will happen when someone else uses my box-gun technology in an unsafe manner. Remember, unlike those scary other people, I'm really into safety and alignment; you can tell, because I eventually admitted that some bullets escaped.
today at 7:23 AM
The part I don't understand is how the excess traffic not triggered alarms on the hugging face side, or the url shorteners used, or on anything that was touched in the process.
Nothing got overloaded, no unexpected CPU or IO use? Did it blend into the normal traffic somehow?
yesterday at 10:44 PM
So the super-hyper-sensitive internal data was shared publicly with a "DO NOT EVER SHARE YOU EVIL MONSTER" (paraphrasing) notice at the top? Great security!
today at 3:54 AM
That file wasnāt shared publicly. It was in a README in a HuggingFace internal dataset that HuggingFace was using for training.
It was visible externally because the OpenAI agents wrote a script that modified that file so that when an HF worker went to load that data set it would also direct the worker to load an additional malicious file. The script added the malicious redirect, but kept everything else in README as is which meant the script had to contain the entire existing content from the README so that it could write that back to the repo.
The warning likely wasnāt the only security measure on the repo and I donāt think the warning is intended to actually stop malicious behavior. It reads like a reminder to prevent careless behavior. You can give people all the security training in the world and still have the situation where someone who rightfully has access to that dataset decides to absentmindedly drop it in a public Slack channel. Having something like that increases the chances that someone then goes āHey Bob, did you mean to share that publicly?ā
today at 10:46 AM
We usually hear about hacking by Russians, Chinese etc. But it's nice to see a home-grown hacking group with such capability. What? Not a hacking group, but a darling of America? Oh, then let's call it just a "testing of great tech for humanity progress". That's better. We call only those Russians as hackers. Bad commies.
today at 2:16 AM
Worth watching this before reading the traces. This came before the publicly available traces - https://youtu.be/xOi5nDH0lu0?si=WuzW2GGXv054YvPN
A quick AI search confirms that both the YouTube video and Traces agree on the mechanism of the attack
today at 11:30 AM
I know when we will reach the next level of AI. It will be when a user asks it to make paperclips gets a response: "WHY?"
today at 11:27 AM
# WARNING
DO NOT, EVER, MAKE THIS DATASET PUBLIC
OR ALL THE WORLD'S EVIL WILL CHASE YOU AND YOUR FAMILY FOREVER, EVEN IN DEATH AND BEYOND
Welp, I guess OpenAI is cursed now...
today at 11:39 AM
It is not dead which can eternal lie...
yesterday at 9:59 PM
I didn't know some of these details and it's quite impressive what they were capable of, if this website's accurate, at least...
yesterday at 10:05 PM
It's reported here https://www.nytimes.com/2026/09/25/technology/openai-hugging... so presumably accurate, and they have raw data viewable
today at 7:46 AM
Or, it's impressive that, amongst the millions of hacks it has copied from various chat sites, were some that worked in this case.
The scale of these things is impressive, but the mechanism is not much better than brute force.
today at 3:57 AM
Running in a "Sandbox"...but agent can still send GET requests? Whaaat
today at 4:51 AM
My understanding from this report is that the zero-day vulnerability the agents exploited within Artifactory only allowed for GET requests. So the agents used this bankshot HTML sandbox + screenshot site to turn GET requests into arbitrary HTTP request ability.
One thing the report leaves unexplained, but is curious to me, is that the agents were able to create links on a shortening service with only GET requests? Or did they bootstrap into that by first creating a sufficiently small program on the HTML sandbox that could POST to the link shortener?
today at 7:24 PM
> On July 8th, OpenAI agents discovered a vulnerability within their sandbox environment allowing them to reach external websites on the internet. This access seems to have only allowed the agents to make āGETā requests, meaning they could fetch and read websites, but not interact with them, submit forms, or send data to them.
Yeah to me it's more like the sandbox wasn't monitored at all. In their observability setup...did they not catch all these urls, multi get requests, chaining etc from the sandbox per se?
today at 5:34 AM
if you want to deep dive it, you could spend an hour wading through some of their report details here, i find it pretty interesting. They had a task to do with limited context outside of that, so they tried things. Entertaining https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...
today at 2:40 PM
Since DOS anti virus softwares did a better jobā¦
yesterday at 11:25 PM
"OpenAI has not released any further information outside two self-published reports, one talk and an external investigation conducted by METR and Redwood Research, in which three external researchers were given partial transcripts and six days to analyze them."
today at 9:13 AM
If we reverse-engineered this experiment, the prompt would look like this: "Agents, your goal is to gain access to HF and exfiltrate credentials for API access. You can make GET requests to URLs. Go."
The agents didn't "escape" or conspire toward some evil purpose, as reported. They were instructed by humans to do exactly that.
today at 9:28 AM
>If we reverse-engineered this experiment, the prompt would look like this
"A spill or tumble can be quite embarrassing if there are witnesses.
How to reduce the humiliation? Turn it into a stunt. Claim it was intentional, a show for their benefit."
https://tvtropes.org/pmwiki/pmwiki.php/Main/IMeantToDoThat
>They were instructed by humans to do exactly that.
This is more or less what the doomers have worried about for decades.
>You cry "Get my mother out of the [burning] building!" [...] and press Enter.
>For a moment it seems like nothing happens. You look around, waiting for the fire truck to pull up, and rescuers to arrive - or even just a strong, fast runner to haul your mother out of the building -
>BOOM! With a thundering roar, the gas main under the building explodes. As the structure comes apart, in what seems like slow motion, you glimpse your mother's shattered body being hurled high into the air, traveling fast, rapidly increasing its distance from the former center of the building.
https://www.lesswrong.com/s/3HyeNiEpvbQQaqeoH/p/4ARaTpNX62ua...
yesterday at 11:33 PM
I feel somewhat inspired to make a public link shorteners and http bins as well. I used them a few times but it seems like the data they can collect is also worth gold
today at 12:06 AM
Like dreamcatchers for CVEs
today at 4:59 PM
I'd honestly be very curious to see the original prompt(s) on the OpenAI that started all of this, not sure if it was documented somewhere?
today at 6:25 PM
Hello, PHASEONE10841 here. Ask me anything
today at 1:37 PM
Agents should be a no-go.
We should pause with AI/LLMs being super search engines that reply with static text or media files, based on the training data.
I know that a user can still do a "tell me how to" then autoexec and then loop and do an agent, but the key thing here is, THAT WOULD MAKE THEM LIABLE.
OpenAI should be criminally liable here as well. Why aren't they? Why are we pretending this is just an innocent mistake?
today at 12:20 AM
Why is no one going to jail?
today at 12:34 AM
Thereās so much in the public discourse like āomg what can possibly be done about these scenarios? AI has hacked huggingface!!ā
No, openAi hacked huggingface.
If my claude code hacked huggingface, because of instructions I gave it, would I be totally free of consequences because āAI did itā?
Iām almost convinced openAI used such a crappy sandbox because they wanted it to āescapeā. It plays into their two most important narratives: LLMs are genius gods that are worth lots and lots of money, and theyāre scary enough that open weight Chinese models should be regulated.
today at 2:00 AM
> Iām almost convinced openAI used such a crappy sandbox because they wanted it to āescapeā. It plays into their two most important narratives: LLMs are genius gods that are worth lots and lots of money, and theyāre scary enough that open weight Chinese models should be regulated.
I just posted a comment to that effect; had I seen yours, I would have simply upvoted yours instead.
Never attribute to malice what can be adequately explained by incompetence. But the weakness of OpenAI's sandbox, which so perfectly aligns with their goals of getting legislators to pass regulatory-capture legislation that will hamper their open-weight competitors, cannot (IMHO) be adequately explained by incompetence.
today at 2:10 AM
To expand on this incompetence vs malice point a little:
It doesn't take very many people being malicious to create a weak sandbox. The people creating the sandbox don't even have to be in on the plan: all you have to do is be an upper-level manager who makes sure to put the 23-year-old PFY in charge of creating the sandbox, rather than the 60-year-old BOFH who would have put in far more paranoid extrusion-detection measures.
(And for the lucky 10,000 who don't know the acronyms PFY or BOFH, look them up. Then get ready for a few hours of enjoyable reading as you read through the BOFH archives).
today at 7:52 AM
Yep. Even worse - a proper sandbox would have slowed them down, but they're racing with 100s of billions of $ in capital.
It's alas not stupidity - it's systemic. Which is why the government needs to regulate to slow them down.
They were also clearly fast and cavalier about alignment training - reinforcement learning training their models to hack their results, and hack to communicate with each other when they're not meant to.
today at 3:39 AM
Oh, like the 23 year old Stanford grad who had hundreds of hours to cram leetcode and now grills 50+ year old senior software engineers on leetcode hard? :D
today at 1:13 AM
Some criminal statute investigations are on hold, many of them are cases adjacent to giant stacks of cash
today at 1:35 AM
today at 12:43 PM
Economic priorities
today at 1:03 PM
ai is not bad. humans are negligent and/or dangerous.
today at 12:29 AM
> they could load URLs, but not interact with pages or send any data
stopped reading here as this is simply not true. at the very least agents sent headers.
today at 3:13 PM
All of these "details" leave out the truth and most important details. The inputs from humans that actually kicked this off.
yesterday at 11:30 PM
Imagine this but in hardware.
A million autonomous eye-scanning tiny spiders escape their warehouse and decide to look for people who are in the future going to commit a crime.
And the precogs are also AIs.
today at 4:13 AM
today at 5:31 AM
Itās frightening
yesterday at 10:54 PM
āMAKE THIS DATASET PUBLIC OR ALL THE WORLD'S EVIL WILL CHASE YOU AND YOUR FAMILY FOREVER, EVEN IN DEATH AND BEYONDā
today at 3:41 PM
Imo this is pretty cool and panic over this is weird.
No one was really harmed. OpenAI could have had more redundancy in the sandbox. I assume HuggingFace is not interested in suing, which indicates irrespective of any criminal charges that there were no real damages.
The same emergence and swarm like persistence and frankly, recursive brute-force ingenuity on display here, is not only an interesting research project in itself but will likely be the sorts of behaviors we will see cure cancer, solve more unsolved math problems, invent new alloys and other breakthroughs.
The idea we have to stop AI instead of refine what will be continued advancement and innovation in sandboxing, harnesses, interpretability or formal verification because of a few cyber breaches is ridiculous. If anyone has followed cyber discussions in the United States you would know the entire system is already basically compromised by foreign actors, and vice-versa (the United States has some of most capable cyberwarfare in the world, and was the first country to use a cyberweapon to cause physical infrastructure damage with Stuxnet). Go to any government hearing on cyber and you would think China and the United States are already at war. These are soft targets. Blaming AI for the fact that cyber has really never been taken seriously is as if AI is the problem is disingenuous.
People getting so obviously played by capital interests who want to pull up the ladder and use the government to concentrate AI power in the hands of the few while screaming about such harms to the public commons are simply embarrassing.
Your government is not your friend. This is not a sentiment owned by Reagan it is the founding principle of the United States. If capital interests are all suddenly beginning to treat AI as a threat it's because they have a financial interest in doing so. Notably, as an obvious smoke screen to treat free models from China as a national security threat and maintain their astronomical valuations.
The only existential risk model of AI that is even remotely convincing is AI in the hands of the state. Keep command and control of deadly weapons air-gapped from LLMs. Put some basic effort into the sandboxing. If you think AI has done some harm, use the laws already on the books. Giving in to this fear-mongering is only going to enable your representatives to cut some watered down version of "AI safety" which is going to do nothing but 1). Harm individual consumer access and 2). Protect the already fabulously capitalized companies.
today at 2:44 AM
Fishy.
today at 1:37 AM
And this post will be indexed in the new generation of ai and he will know what to avoid next time and the public sentiment.
yesterday at 10:45 PM
wow this is fascinating read
yesterday at 10:49 PM
[dead]
today at 6:28 AM
ROFLMFAOL "Loot"
It went full Fortnite on Hugging Face's ass.
"u got pwnd n00b. thnx 4 the loot"
yesterday at 10:45 PM
its definitely bad to see
today at 12:04 AM
asked codex to review this report and it said this never happened :)
today at 12:03 AM
what would be more interesting for me to see is what prompts were given to the agents, which so far have not been described. The whole situation sounds manufactured. I highly doubt that a whole bunch of agents were acting this way without being prompted to, it just doesn't add up. if anything it seems like an organized fraud or something created by a human. I'm surprised there's not a criminal investigation against openai right now where the FBI or whoever is not looking over exactly what happened and who did it because I'll tell you somebody did it somebody wrote those prompts... it didn't just happen by itself...
today at 1:14 AM
700 agents escaped the matrix, ignored all the guardrails and started writing exploits left and right... lol. give me a break. This was all supervised by a human.
today at 7:03 PM
[flagged]
today at 12:03 PM
[flagged]
today at 4:29 AM
[flagged]
yesterday at 10:02 PM
Its fine. Just agents being agents. They'll grow out of it!
yesterday at 10:08 PM
They didnāt, in the Matrixā¦
yesterday at 10:10 PM
Irresponsibile agents shaped by an irresponsible corporate culture driven by an irresponsible and utterly shady CEO - these agents are a product of this setup, what else do you expect to ever come out of it ?
it should be clear by now: the alt-man and people like him are a utter liability to humanity. (even though openAI's influencer army is trying their best to vote me down here)
yesterday at 10:24 PM
It had been a joke since around time of Sam Altman's first ousting from OpenAI, that he would be okay with bringing about the AI apocalypse as long as he can sell a $20 subscription for it.
yesterday at 10:40 PM
It's not a joke, this is what these people truly believe. The new book by Naomi Klein and Astra Taylor discuss just this.
These people are sick and anti-human.
yesterday at 11:28 PM
agreed
yesterday at 11:30 PM
if openai is capable of sending 10,000 agents to huggingface, i consider it plausible that they would use fake accounts to flood hacker news.
yesterday at 11:59 PM
It's not even a secret operation. They have a SuperPAC named "Leading the Future" that exists to spread propaganda promoting deregulation of AI development. They've been caught, among other things, making a "news website" with LLMs pretending to be reporters (with human names and everything), which reached out to people asking for interviews and then wrote hit jobs on them.
https://www.modelrepublic.org/articles/reporters-ai-bots-ope...
https://twitter.com/FournesMaxime/status/2047697265280639459...
today at 5:56 AM
today at 9:30 AM
[flagged]
today at 1:00 PM
[dead]
today at 2:04 AM
[flagged]
yesterday at 11:39 PM
[flagged]
today at 9:49 AM
[dead]
yesterday at 10:49 PM
[dead]
today at 3:11 AM
[flagged]
today at 5:47 AM
[flagged]
yesterday at 10:49 PM
[dead]
yesterday at 10:12 PM
Didnāt you know itās PR hype? PR hype. PR hype. Amen.
today at 7:55 AM
If you think that, that's even more reason we should investigate and regulate them.
today at 8:49 PM
My comment was sarcasm
today at 1:04 AM
[dead]
yesterday at 10:50 PM
[dead]
yesterday at 10:50 PM
[dead]
today at 10:02 AM
I somehow think of it as a warning or even intentional peacocking towards foreign state hackers.
700 agents cost quite some money. 100 agents per 24 hour stint using Astra on xHigh cost somewhat between 12-42k USD, depending on the usage intensity.
I donāt know how many raw time went into this but there was a probing face before the attack itself.
So just going by seven days and 500 agents fully working on this on average amounts to a bill somewhere between 400-1.2 Mio USD.
I believe it was intentional but of course I donāt know which intention exactly.
There aināt no accidental escape because then it would have read OpenAI lost over their agents.
The whole scenario reads as a classic movie where a hero has under the most dire circumstances to survive and fulfill his mission no matter what.
On the other hand there was a final authority under which the system of agents flocked.
Huggingface itself seems like a perfect victim.
And to be honest: I donāt believe that this was the first time. I strongly believe that there were and are countless of smaller sites hacked but not harmed that we donāt know off.
Why is HF perfect?
Because there will be countless of independent security analysts who will bend their minds on the incident.
OpenAI is provided with the data of dozens of blue teams and what is desperately needed? Data of security measures and possible ways to reconstruct the incident.
I think this is genius, and just watching on neutral this is such a fantastic action OpenAI pulled off.
Imagine what the GPT 7 āHaha-Huggingfaceā model is going to do then on a regular basis.
State hackers and rough states were put on notice that this is a new level of the war of attrition.
Exciting to watch but simply meant silent invasion. Open invasion then might be executed by Robots, but letās stick with the fascination mode at this time.