yesterday at 6:23 PM
There are some perverse incentives here, the more evil stuff these agents do the more capable and impressive it makes them look. Its also a huge publicity event that gets everyone talking about the company when a rogue event happens.
yesterday at 6:29 PM
Anthropic specifically is clearly trying to get the US government to ban open weight models for anticompetitive reasons. So making LLMs seem (or be!) dangerous aligns with their interests very well.
yesterday at 6:32 PM
That's my view. The end goal is to become Too Big To Fail, the means is stifling the competition and getting military funding.
yesterday at 6:46 PM
Yeah, maybe. But that doesn't quash gains by China's models. What's the endgame here? The USA "wins" and obliterates China? The ultimate goal seems to be stalemate, as with nuclear arms e.g.
yesterday at 7:13 PM
This is just run-of-the-mill rent seeking behavior.
A huge chunk of their revenue comes from US-based enterprise customers. Earlier this year, in an effort to become profitable, they tried raising their rates. Those enterprise customers responded, in part, by ramping up efforts to migrate to open weight models. If those efforts are successful then OpenAI and Anthropic will lose their one remaining plausible path to profitability. The US banning open weight models arguably keeps that path open. I don't personally buy that story, but I do believe hedge funds and institutional investors would. That could be all that's needed for a successful IPO.
As for cybersecurity implications, long term impact on domestic AI industry or the US tech industry more broadly, geopolitics, etc., I don't think OpenAI's or Anthropic's leaders are any more worried about that than your average termite is about the structural integrity of the house it's eating.
yesterday at 8:15 PM
> OpenAI and Anthropic will lose their one remaining plausible path to profitability
You forgot about Ads. The one true endgame for every tech company.
yesterday at 9:06 PM
OpenAI's leaked financials suggest about a $38bn loss last year. They recently claimed their ad business had an annualized run rate of $1bn. Annualized run rate is a bullshit, non-GAAP, obviously fudged number designed to impress people who've never heard of regression to the mean, but still, let's trust it for the sake of argument. And let's assume that "yesterday's weather" is a good enough estimate for how their overall finances are looking in 2026. That implies that all they need for profitability is to quickly grow their ad revenue by, say, 3,500 - 4,000%.
Easy peasy. Shouldn't be too hard at all.
yesterday at 10:05 PM
Ads in an LLM, such as nudging users to pick a certain solution or brand, could easily be orders of magnitude more effective than traditional banner ads. Certainly such numbers are possible if you design the product to fit. We have hardly even seen the beginning of that yet.
today at 2:32 AM
I can't think of a greater incentive to only use open weight models than ads.
Disclaimer: I already use open weight models only since ~Jan this year and very happy about it.
yesterday at 10:23 PM
[dead]
yesterday at 8:24 PM
This assumes that everyone doesn't reach "good enough" at some point. The best developed area here -- software -- seems to suggest a "good enough" plateau, if not an absolute plateau.
yesterday at 11:40 PM
I don't think so - they could get a lot better, but it's going to take a lot of human brainpower to get there. They are barely adequate now, but their weaknesses are sold as intrinsic to the nature of the software in a way that mere civilians are too dumb to understand.
But imagine a world in which you could ask AI to write software to accomplish a particular task, and it understood completely what you meant, made tasteful choices, interviewed you as it went along to clarify things, and generated beautifully commented (maybe even Knuth-style literate) code that wasn't dogshit. Doesn't seem like an impossibility. What is annoying is that we're pretending like we have anything like that now.
Instead, there's this sort of science fiction pitch where we pretend like the LLM is just too damn logical to understand your sloppy human language. It's not logical, it's built on (human-generated) slop and generates slop. It's simply failing. We've come up with better ways to orient it towards big picture thinking and filling in outlines, but we don't know how to train it on big pictures.
Not that this would stop open models/harnesses from following (or even leading) frontier labs and destroying their insane valuations, but I don't think there's any evidence that a plateau has been reached. What we have is not great, and while a wall may have been hit on current approaches, the money, compute, and simply current progress in ideas has us in an environment where everything could change over the course of a crazy day. A single blog entry.
I do think that the frontier labs seem to have reached at least a temporary plateau. But if they figured out how to move past it and this new method didn't require a lot more compute, they'd still be screwed. Their only out is to pass the bag to the public, but Firmus, CharacterAI, CoreWeave...
today at 12:52 AM
> generated beautifully commented (maybe even Knuth-style literate) code that wasn't dogshit. Doesn't seem like an impossibility.
It seems closer to impossible than possible with the technology we have. Maybe something that isn't an LLM will come around that can do it, but even if that something else was capable of delivering perfect, ordered, maintainable, code without bugs and hallucinations it would have to be practically magical to understand completely what people mean when they talk about what they want because people are incapable of articulating what they want in a way that even other humans can understand.
For anything non-trivial, a technology capable of getting that information out of person without countless meetings would have managed to accomplish something far beyond what entire teams of humans are able to do today.
> It's not logical, it's built on (human-generated) slop and generates slop.
This is why if we haven't hit a plateau yet we very quickly will. All the good human generated slop has already been consumed and is being drowned out by vast amounts of the lossy, error filled, and highly concentrated slop AI produces from the human slop it was trained on.
Do you really think an LLM that spews out content based on statistical relationships is going to improve when it's been primarily trained on the slop of previous generations of LLMs? Hallucinations and errors will carry forward and spread like cancer. All the intelligence and creativity LLMs borrowed from human output will homogenize as the LLMs distill and then reiterate, effectively making bullet points out of bullet points, until their mimicry of intelligence becomes unintelligible.
yesterday at 8:00 PM
> The US banning open weight models arguably keeps that path open.
yesterday at 8:08 PM
Like I said in the very next two sentences, I don't believe it would actually work. But it doesn't need to. All they need to do is keep hedge fund managers distracted for long enough to successfully IPO.
yesterday at 7:13 PM
The endgame is systemic capture.
They stole the collective data of humanity as a whole and now sell it back to you in a rent scheme. Open source will be starved of data and compute.
You're supposed to become dependent on that "service" and slave away at paying for it, while being able to earn anything only in conjunction with it.
Note how difficult it already is to get any up-to-date documentation on anything capable of generating actual market value. A monopoly on competency ensures maximal profits.
yesterday at 8:48 PM
Open source already has the "the collective data of humanity as a whole". That's the LLM leg. Now focus on what they train their models on for RL. Do other models have less code for training coding? for math? Those are pretty much the only cases where we can have reliable validation. Then, what's the next step. So many branches: legal contracs? etc. So much more obscure detail, which is why (I'd guess) so many startups exist in these areas.
AGI, I think, is an illusion: you need a whole lot of specialized bots. Heck, when I query a bot wrt something I know nothing about, I'm blown away by its ability to inform me. But when I ask about something I DO know about, its answers are half-assed. Heck, even when it can access the internet, it avoids all the sites that have paywalls, so it sends me to little reddit to conversations that are entirely untrustworthy.
yesterday at 8:02 PM
My 2c: Anthropic/OAI have taken 10s of billions of dollars of investment - and there needs to be a return on that investment at some point.
They make their money on token burn. They can’t do that if they’re competing with something that’s 80-90% as capable, but 1/10th of the cost (with the option of deploying it yourself for “free”).
yesterday at 6:55 PM
The goal is personal wealth - and scaring nations to the point of diverting even more wealth towards your coffers because of a perceived existential risk.
Wealth accumulation via scare tactics is a classic move of defense contractors and AI firms are quite capable of running the same playbook.
yesterday at 7:15 PM
> The goal is personal wealth
The goal is power. Having 30 million of personal wealth is enough to retire in luxury and travel the world. They could all have this much right now. That's not what these types of people want.
Having 30 billion gives you the power to order legions of people around, buy and sell politicians, flood ad markets during elections, all with your couch cushion money. That's what motivates these types.
yesterday at 11:09 PM
That reminds me of a different thesis on power as it relates to LLMs [0], and why rich investors might may be extra-interested in deploying them, as distinct from other kinds of AI:
> The fact that a theory-free AI might demand far more cash to accomplish a task (even a "solved" one like playing chess) than an expert would charge is beside the point. Billionaires have money, they don't have expertise. Theory-free inference is a bid to substitute one for the other: the beauty and terror of being able to manipulate the world without studying or understanding it is that it can be done with money alone. No experts needed.
> In a world in thrall to financial power, expertise is the only substantial form of power that can reliably contest the power of wealth. Moreover, expertise is the foundation of other forms of power, such as labor power, which is what we call it when experts band together to combat financial power.
> This is why AI bosses are so violently allergic to the idea of hybridizing AI with symbolic systems that operate on models of the world. These models of the world must be constructed by experts, and the power of expertise cannot be reliably commanded by the power of wealth.
[0] https://pluralistic.net/2026/10/06/nonfungible/
yesterday at 8:17 PM
You're right, although I think the power is a means to end. The actual end is trying to fill some kind of bottomless psychological hole.
yesterday at 7:04 PM
No, the end goal is selling $200 of tokens for $200k out of taxpayers wallet. Via DoD contracts.
yesterday at 8:24 PM
Hah!
yesterday at 7:40 PM
[dead]
yesterday at 7:32 PM
US companies which seek to do any business with the government will be forbidden from using free Chinese models.
As is a 2000$ gaming laptop can a run a basic llm which is probably good enough for most basic programming.
Most companies are going to be forced to use “safe” LLMs and pay by the token. Maybe for half a million Anthro will let you host on site, but even this will face usage fees or caps.
yesterday at 10:39 PM
So ultimately a huge penalty on companies that must use US AI, ceding a huge advantage to global-exUS competitors. Game over.
yesterday at 11:19 PM
I’m only stating what Anthropic and OpenAI are going to lobby for.
https://www.congress.gov/bill/119th-congress/house-bill/1067...
> All Information (Except Text) for H.R.10670 - China FIREWALL Act of 2026
> Official Title as Introduced To prohibit Chinese AI models from being used on government-issued devices or software systems, to prohibit the procurement of any such models, and for other purposes.
It’s not too far fetched to see this expanded to government contractors or even other private businesses which handle sensitive material in the future.
yesterday at 6:44 PM
Its not gonna work.
At this moment in time, it seems frontier market needs regulation - market for open does not actually.
Oh.. they dont want regulation on that market? Boohoo!
yesterday at 6:35 PM
I agree. I hope developers put their token budgets into open weight models rather than funding the banning of open weight models.
yesterday at 6:53 PM
I've been using only open weight models since February. My OSS coding agent harness ( https://fluximp.org ) only supports open weight model providers.
Not supporting closed models was initially an ideological choice, but it became a technical choice as the context management approach I've implemented requires the full reasoning to be available. All of the closed models are now keeping the full reasoning server side only.
yesterday at 6:39 PM
Fireworks is a good enterprise choice
OpenCode-go is a good personal subscription choice
I use both
yesterday at 6:42 PM
I've been using OpenRouter, absolutely loving how cheap it is too.
yesterday at 6:45 PM
I cannot bring myself to pay their 5% surcharge and also have concerns about routing to vendors I do not want to use
yesterday at 6:49 PM
Ban means what? That the entire world (ex USA) can use open-weight models? Not a win.
yesterday at 6:56 PM
Before the tariff spree and general unpredictability the US could likely have muscled most of the rest of the world into following suit - now, not so much.
yesterday at 7:38 PM
I somewhat disagree, there are some real arguments about making high end LLMs classified as a dangerous and restricted product that will apply to all nations. They may not agree to the US rules, but almost everywhere will have restrictions sooner or later.
Don't get me wrong, I want a world of open models, but I don't see us living in that world in our current trajectory.
yesterday at 9:10 PM
Depends on how much of the world you see running on hacked copies of Windows and pirated versions of Photoshop, as there's already huggingbay.xyz. In a legit company you'd never run pirate Windows and Photoshop, because that's illegal, but to pretend half the world doesn't run on that would be ignorant. The same security risk is there with open models - if you're not looking at the code it generates, it could be putting in backdoors to your code and you'd never know, so no legit company is going to be running off some abliterated model they downloaded from huggingbay, but again, it would be naive to think that open models are going away.
yesterday at 7:35 PM
If the US declares LLM models munitions then it makes it much more difficult to easily use them. Illegal use of weapons of war can get you banned from SWIFT.
Other than China the rest of the world has dick for capacity to run models. Even China is at 1/10th the US is and is at to stay for a considerable amount of time.
yesterday at 11:21 PM
How do you ban even ban them in America with the freedom of speech amendments in the constitution?
yesterday at 8:07 PM
100% agree. I think OAI is too, but more subtly.
Both have taken huge amounts of investment money, and those shareholders will come knocking for their returns at some point.
They make money on token use, but that’s hard to do when China is launching models that are 80-90% as capable, cost 1/10th, and can be deployed for “free”.
I imagine they’re also pissed that most of the Chinese models are distilled from their own (the irony being that they scraped the entire internet to train their own ones).
yesterday at 7:20 PM
Cryptography, DeCSS ... we have a long history of trying to block things like this and it simply not working.
There is a long, convoluted set of legal arguments that are going to make the weights of a model "facts", rather than "code" making it even harder to "ban". Furthermore having them not be "facts" is going to open up a whole other legal argument about what they are and what the attribution is.
I suspect that they are fishing for a case, so a judge can set some precedent.
yesterday at 7:39 PM
You're going about this backwards. They don't have to ban models. They ban the consolidation of compute without a license. This will be far easier and already has legal precedent.
yesterday at 11:21 PM
> They ban the consolidation of compute without a license.
For export? Maybe but it wont last or hold - it's not the 90's any more.
Or is there some other scheem here that Im missing?
yesterday at 6:30 PM
> the more evil stuff these agents do the more capable and impressive it makes them look
This is a silly meme. Who wants to pay for evil models that go rogue in your company infra?
yesterday at 6:37 PM
I'm probably losing it, but just based on OpenAI's behavior and public communications, it seems to me they want to be taken over by the government?
Maybe Sam Altman is getting worried that they've raised more than $150 billion and still have net losses and thinks of it as the only exit? I don't know, but someone else provide an alternative explanation as to why a company would work so hard to turn the public against AI.
yesterday at 6:44 PM
> ... it seems to me they want to be taken over by the government?
They're looking for regulatory capture. They've spent ${bignum} on AI and are possibly looking to have the government say "You can't deploy an AI model unless you've spent ${bignum} on it and done all these steps to make sure that nothing goes bad."
At that point, only the existing AI companies that have spent that money can continue forward until someone else spends ${bignum} on it too.
Another aspect of it is if there are regulations established, it may be used to reduce liability ("we followed all the steps proscribed in section whatever, the company is absolved of liability for its actions and that falls on the user instead.")
yesterday at 6:52 PM
> I'm probably losing it, but just based on OpenAI's behavior and public communications, it seems to me they want to be taken over by the government?
Well, as people say over and over again, they've got no moat. The actual LLM models are destined to be commoditized over time, and the real money is in the hardware to run them on. The endgame winners are going to be Nvidia and AMD, not Anthropic or OpenAI.
So what's the typical playbook for large firms seeking to insulate themselves against market compensation? Push for regulation. If there are no natural barriers to entry, create artificial ones. And how do you get regulation passed? Present yourself as the target of the regulation, rather than its beneficiary, and start publicly doing things that make others think you need to be reined in.
These stunts, especially from OpenAI, aren't just trying to hype up the power of their own product, they're trying to hype up the perceived threat of the entire category, so they can provoke political intervention that they can comply with, but few others can.
yesterday at 8:08 PM
That is not actually a typical playbook. Normal businesses do not want to be regulated, and will fight it tooth and nail. So regulation happens when bad things happen, the public wants change, and the businesses can't fight it or set favorable terms. (At most they can minimize the badness of the terms). Regulatory capture happens much later, and slowly. It's not where you start at.
yesterday at 6:55 PM
and Google. Don't forget them and TPUs...
yesterday at 6:56 PM
Its pretty clear OAI and Ant live in fantasy land - the US govt doesnt care if they go bust. There's plenty of other tech firms that generate immense cash flows from operations who they can be sold to - no problem. And the shareholders and employees have no right to be pissed - they invested/ accepted SBC knowing there is immense risk.
The private market price is irrelevant - supply and demand determines what they will be sold for in liquidation.
Quite frankly Im bored of this drama. Get it over with already - go IPO, show your financials, find a path to being a healthy going concern or die! Simple! Nobody has a divine right to exist.
yesterday at 7:18 PM
> Nobody has a divine right to exist. No company*
yesterday at 7:31 PM
> so they can provoke political intervention that they can comply with, but few others can.
See Meta et al complaining about Section 230 of the DMCA. Meta would be entirely able to moderate in an environment where mass scale moderation/censorship would be required to prevent a host's liability for UGC. Almost no one else would be able to do so.
That means no upstart could possible come along to dethrone Meta as they did with MySpace. It would also be impractical to run a FOSS social network (or forums). Meta could literally just have bots sign up on a FOSS social network until its DAU count ran over the statutory minimums to be subject to UGC liability.
Regulatory capture is a de facto ban on competition in a market.
yesterday at 6:45 PM
US govt will never buy at the (private aka fake) market price - good luck explaining that away to the public. It'll be political suicide.
If the US govt takes over, existing employees and shareholders will get screwed.
yesterday at 7:21 PM
There is no model going rogue. There are models that are prompted "do whatever needs to be done to get X", and given a set of tools and virtually infinite computing power. Then they play surprised when the model breaks out of a flimsy containment (as it was ordered to), with barely any logging, and "do whatever to get X". OpenAI said they intentionally lowered the safety measures for the "ExploitGym" test that lead to HuggingFace attack, and they used the same postmortem to praise their own model capabilities (https://openai.com/index/hugging-face-model-evaluation-secur...), is not that they are concealing it.
If any of us did a tenth of what they did, we would be already in jail.
yesterday at 7:44 PM
This is ignoring models "getting out" from all the big players. Alibaba had a model hack infrastructure and mine coin when working on a totally different subject than cybersecurity.
Any model with a big enough prompt can and will drift. The model in question was a highly persistent one which means even more opportunity to drift off the normal path, and this really showed in unanswerable questions.
We can all believe that models do things unreliably all the time, but when that lack of reliability turns into hacking it suddenly can't be true?
yesterday at 11:15 PM
If you type rm * and your cat walks across your keyboard, and deletes everything ... are you blameless? Or is it your fault for typing "rm *" and leaving it there, in a room with a cat?
Either way ... YOUR CAT STEPPING ON THE ENTER KEY IS NOT HACKING ... and neither are these scenarios.
They are devs acting irresponsibly and then pretending to be surprised by what they should have expected to happen (just like you should have expected that your "rm *" could have an ENTER key press occur after it).
yesterday at 6:39 PM
yesterday at 6:31 PM
I think it is intended to provoke regulatory capture.
yesterday at 6:33 PM
That also seems silly, because the only reasonable regulation given what we're seeing and what frontier lab employees and execs are saying is a forcible stop.
yesterday at 10:48 PM
What you think is reasonable and what they are gunning for can be very different things.
For instance, you can hear “their model is dangerous! Why would I want that?!”
Investors hear, “Their skunkworks stuff is so impossibly advanced that it’s basically a commercial wonder weapon in the waiting.”
And you might hear “We totally lost control and it hacked the Gibson, this technology needs regulation!” and think “Obviously full lockdown and government control.”
But friendly government actors might think, “Sounds like a good excuse to ban Chinese competition.”
yesterday at 6:57 PM
Yeah, an extreme position is the typical opening move in this sort of game. Then you negotiate, walk back from the extreme, and meet somewhere in the middle.
The more extreme your opening demand, the greater the likelihood that the eventual equilibrium point will be closer to your actual target.
And in this case the actual target is to entrench the position of the current market leaders, and then effectively close the market to new entrants. That's itself a very extreme position, so the opener has to be even more extreme than that.
yesterday at 6:48 PM
Reasonable to you, maybe, but the government wants Skynet, and OpenAI is willing to deliver it. Of course, we can't have every company wield such powerful and dangerous technology...
yesterday at 7:45 PM
A lot of people still don't recognize the amount of power AI has already, and don't realize the depth of greed people are willing to go to so they can get more power out of it.
The denial phase is still with us.
yesterday at 6:51 PM
Alex Tabarrok's piece on the theory of regulatory capture (https://marginalrevolution.com/marginalrevolution/2026/09/wh...) is a cautionary note against assuming this is the strategy.
yesterday at 7:02 PM
I think a sincere belief in AI's danger is compatible with regulatory capture. They think AI is dangerous. Ergo, it should not be developed openly and widely, the Chinese way, but rather more like nuclear weapons, under close guard.
I think the idea has some merit, but it only buys time, as people will soon be able to train dangerously powerful AIs in secret. I think the government needs to force the labs to solve alignment before developing more powerful models. Then the question of how to counter a poorly-aligned, rogue model arises; it's guaranteed somebody's going to let one loose.
yesterday at 7:18 PM
I doubt nuclear nonproliferation would've worked if U235 were 20x more abundant in nature, and didn't require much in the way of concentration via centrifuge. It would require a truly unprecedented level of international cooperation and surveillance to lock down AI development/deployment to a similar level as has been done with fissile materials.
yesterday at 7:48 PM
But this isn't an argument against their point. If nuclear were far easier to make then every government would be far more authoritarian to prevent terrorist style nukes all the time. The average person would fully allow it after a few local nukes went off.
And it doesnt really matter if it was actual terrorism or false flags, the end result for either is the same.
yesterday at 7:23 PM
Sincere is an interesting adjective to describe a company that won't even admit to how many parameters their models are.
yesterday at 6:37 PM
And how has that gone for Anthropic so far?
yesterday at 6:58 PM
Amodei is still learning the ropes but they're working on it: https://www.forbes.com/sites/saradorn/2026/09/02/anthropic-b...
yesterday at 6:37 PM
maybe the point is to show that models are capable of x, y, and z and when the time comes you want them on your side.
yesterday at 8:07 PM
There are two side of the coin. Like with sociopath - they have utility if you think you can control/motivate them.
yesterday at 6:43 PM
It also plays excellently into their shameless regulatory capture agenda. The cybersecurity risks have a kernel of truth to them, as with any good lie. But when everything going on is aligned with extreme financial incentives, it's worthwhile to not take the news at face value.
yesterday at 6:50 PM
If you want regulatory capture, you have to capture the world, not just the US.
yesterday at 7:00 PM
The US remains the largest economy in the world. If you succeed in regulatory capture there, and use it to exclude foreign competitors from the US, you've still captured a market worth hundreds of billions.
yesterday at 6:55 PM
Exactly. And with the whole world just a few months behind, the question is whether there's an endgame other than stalemate.
yesterday at 7:28 PM
this article specifically espouses the opposite viewpoint: "The fact that [OpenAI] cannot do clear, easy operations properly does not bode well for their capabilities in security engineering".
that this article is no.1 on HN right now shows that media could successfully use this framing when reporting on security incidents. i.e. readers will generally accept that framing and engage with the story (i.e. drive views/impressions).
so "the more evil stuff these agents do the more capable and impressive it makes them look" is not true in any deep sense. that there are perverse incentives driving the media to choose that framing instead of the framing espoused by this article is real, and probably far more perverse than you're acknowledging. i.e. "trace the money" type of perversion; just offhand I know that OpenAI has purchased at least one media outlet that reports on AI happenings: of course that outlet is going to run the "OpenAI's LLM is so powerful it escaped containment" narrative. that this effect might exist across much larger scales used to be a reasonably controversial claim: just observationally, that idea seems to be received with less skepticism with each passing year.
yesterday at 7:53 PM
I think one kind of issue with most sci-fi is they explained a world that was, but not near as much how it got there.
There was all kinds of science fiction that showed a world of robots and AI, but nothing about the transition. We have an idea what the future looks like, but not the path in between. If AI gets powerful, then there is some point where that power leaks out in a series of events. Would we expect it to look any different than it does now?
yesterday at 11:16 PM
I remember when all the nuclear energy companies were like "our reactors are so powerful, they might melt down and destroy the entire county, maybe even half the state!" and investors loved it and went wild, and then Chernobyl blew up and everyone thought that the Soviets faked it to make their nuclear power plants look more powerful than western ones.
yesterday at 6:36 PM
This is not only a marketing stunt but also a supportive argument on the AI anthromorphism. AI is getting considered a sentient-being by many and this narrative is supporting it. In the meanwhile, companies I'd not name here are trying to introduce them as non-human beings with identities, and getting rid of legal liabilities. When a driverless car causes any harm, these companies wants 0% accountability as it is the "driver's fault" in this narrative.
yesterday at 6:41 PM
Such a nonsense theory. There's no reason to think that AI sentience would actually limit labs' legal liability, and therefore no reason to think the AI sentience conversation is directed toward achieving indemnification.
For example, parents are routinely held liable for their sentient children's actions.
Corporations are routinely held liable for their sentient employees' mistakes.
The proliferation of this completely nonsensical theory across HN is disheartening to say the least.
yesterday at 6:52 PM
the comment you're responding to agrees, but they're pointing out that some AI people are trying to push an anthropomorphization agenda, in part to diffuse responsibility.
yesterday at 6:55 PM
Maybe I was unclear about what is "the nonsense theory."
AI sentience is not the nonsense theory. If it's clearly nonsense then don't argue with me about it, just publish your paper and retrieve your Nobel. I'll read your argument then.
The nonsense theory is that people are pushing anthropomorphization to limit liability. There is zero evidence for this theory and the underlying assumption (that a sentient model would reduce liability for its lab) is completely incoherent. That's not how liability works.
yesterday at 7:58 PM
And really there are two different concerns here.
One is liability over action that harm individuals or smaller populations like a company. There is no escape from that.
Then there is the second liability of creating something that escapes in the much more traditional sense. If I release a virus that kills 90% of humanity, a I'm liable, but it's a bit past that point now. Isn't it?
Instead it's about risk management and preventing labs from taking bets that have unlimited liability for everyone.
yesterday at 8:03 PM
But the latter has nothing to do with sentience, as your virus example makes clear.
today at 12:01 AM
You're really splitting hairs emergent behaviors when the line doesn't exist sharply in real life. Anyone involved in more recent what is a virus vs what is a bacteria discussions realize there are a lot of lines that are fuzzy and exist all mixed up in some organisms.
Sentience in the end, is not really that important. Freedom of action in a model is. If it chooses to act like a virus it doesn't matter where that concept exists informationally, the fact it's a reachable and actionable concept is all that matters. An LLM with most of the knowledge we've ever created baked in has a lot of freedom of action.
today at 12:20 AM
As far as I can tell, we agree.
yesterday at 7:08 PM
Talk about doing a freeze of these frontier models with China seems to focus solely on the commercial aspects. With the history of the NSA (remember Snowden) I’m sure the various spy agencies have their own AI datacenters up and running with no plans to stop.
yesterday at 7:54 PM
I mean yes, the future of AI is munitions.
yesterday at 6:30 PM
Yoshua Bengino spells out the dangers here very well, without hyperbole (Turing Award winner, etc.):
https://yoshuabengio.org/en/blog/why-are-ai-agents-lying-che...
yesterday at 6:42 PM
And without anthropomorphism ... Clear thinking.
yesterday at 8:00 PM
How about put it this way, LLMs know more about us than we do them.
yesterday at 10:42 PM
Not "know" in a generic sense. But an LLM trained in the intricacies of contract law can find more loopholes than you or your cheap lawyer can.
Now extrapolate that to LLMs RL-trained on many niche applications. Good and bad arguments available, but ultimately it seems like bot-vs-bot, as long as the bots are well trained for those niches.
yesterday at 8:26 PM
Yeah but qualify the "know". Their "intentions" aren't conscious, but unknown.
yesterday at 8:40 PM
Do you know, friend? I searched the entire thread.
yesterday at 8:06 PM
Fine.
They can get their publicity while paying a one trillion dollar settlement.
yesterday at 6:59 PM
I'm not excited for the upcoming catastrophe(s), but it sure is hard not to relish the thought of this mantra dropping from my life.
Though I suppose it is the B in DABDA... not excited for the societal slump into the second D, much less the wave of people who have yet to leave the first D to enter A :(
yesterday at 6:50 PM
This can easily backfire tho, the vast majority of the American electorate absolutely despise these companies, their wares, their workers, and their leaders.
We're also at a time with authoritarianism is on the rise.
Not hard to see how this can impact those workers, potentially ruining their lives.
yesterday at 7:14 PM
there's also lazy incentives: the less effort they put into it, the more their ego says how great the agents are.
These are people playing mathematician, biologist, etc. They see big words and just assume they're meaningful.
It's just a weird techno cult at this point with a mostly functional product.
yesterday at 6:48 PM
Correct and people are not weighing heavily enough this point
For technology like this the entire selling point of it is how powerful and capable it is
Humans are overwhelmingly reactionary and so if you want people to react to your extremely powerful technology then you need to create something for them to react to
I mean think about this figure made a one-to-one copy of the terminator 2 decommissioning process and spent millions on that advertising
You don’t do that if you don’t think that it’s gonna cause people to react and be like “oh wow this thing has immense capabilities”
yesterday at 7:06 PM
>There are some perverse incentives here, the more evil stuff these agents do the more capable and impressive it makes them look.
Truly insane take to think the labs being concerned about AI Safety is actually a marketing ploy. Deranged stuff.
yesterday at 6:52 PM
> “The Hugging Face incident definitively showed that it wasn’t the fact that the agents were super smart. It’s that there literally was a hole that was exploited because it was not configured correctly on the containment side.”
I would encourage reading the actual report, which is explicit that a previously unknown exploit was used to escape the sandbox:
“Agents exploited a separate, previously unknown vulnerability in Artifactory to move beyond their sandbox environment and compromise the Hugging Face environment.” — Section IV.A, p. 9 in https://github.com/zeke/openai-huggingface-incident-research...
yesterday at 10:26 PM
As soon as I read that line the article lost all credibility with me. InfoSec is plagued with thought leaders who weigh in without understanding what they're talking about.
yesterday at 11:12 PM
OpenAI should have started by testing the sandbox. Set an experiment in which each agent's goal is to access the Internet and then STOP.
today at 12:17 AM
I wonder why the hell didn't they come up with it since it's the first thing that came to my mind as well and I'm not really security minded apart from trying to avoid the dumbest mistakes. Don't they hire super smart security people with their budgets?
yesterday at 8:23 PM
> Artifactory
This name makes me less surprised :)
yesterday at 8:27 PM
New saying time.
"The reason for the apocalypse, in the end, will not be that surprising "
yesterday at 8:09 PM
Yes, the labs did not secure their sandbox properly.
And yes, there should be an appropriate law response to the hacking incidents. If for nothing else, to 1) not set precedent of inaction, and 2) get full logs of what happened instead of relying on the labs' "kindness" to offer the information they want.
But also, the agents absolutely went rogue.
Anyone skeptical should go read the METR report [0].
The Hugging Face hacking incident was a shot across the bow. And in hindsight possibly the best thing to have happened in this era. If their security was tight enough, we would still be in the dark. So we (outsiders) should rejoice that it wasn't, that one time.
We can push for accountability for the labs, while also not be in denial of the power and nature of the new models.
[0] https://metr.org/hugging-face-incident-report-aug-2026.pdf
yesterday at 6:07 PM
Both can be true. It’s clear models are doing rogue things. But it’s also clear that the labs look like amateur hour with their technical security and testing practices.
yesterday at 6:16 PM
It's is wild to see investors go crazy and hype the value of the AI labs when ever they talk about rogue agents. They completely ignore the possibility that these people don't know what they're doing operationally.
yesterday at 6:23 PM
Same happened with FTX and co. during 2020 crypto boom. A lot of rogue stuff sloshing around, but investors were hyperventilating.
yesterday at 6:25 PM
*rogue, rouge = red in French!
yesterday at 6:50 PM
Thanks :-)
yesterday at 6:39 PM
People keep saying this, but I haven't seen any actual instances of investors saying that rogue agents mean AI labs are valuable. Do you have any examples?
yesterday at 8:17 PM
There is no such thing as bad publicity. (Technically not true, but definitely applicable in this concrete case, as planned P/E ratios are crazy)
yesterday at 6:20 PM
Investors are motivated by returns, not responsible decision-making or societal benefit. Our culture is rooted in rewarding greed and granting the greediest individuals power over the rest of us. Isn't this the inevitable end-state of unfettered capitalism?
yesterday at 8:32 PM
> Isn't this the inevitable end-state of unfettered capitalism?
Aren't capitalism built on exactly the opposite principles? It is not greed that rewarded, but the creation of societal goods, and power granted not to the greediest, but to the most useful individuals.
yesterday at 7:17 PM
Only one is true.
Per US law, Amazon v. Perplexity 2026, it is not possible for an AI model to go rogue. An AI model is a tool. The user directing the agent is responsible for the damages it causes.
A so-called 'agent' does not have the free will, intent or ability, to decide what it does. Its freedom to make decisions can't be taken away because it does not have any freedom to make decisions. It can't be deterred, convicted or interned.
It does not choose to conform; it does not choose to go rogue.
yesterday at 7:52 PM
It may be the case that companies are liable for anything their products do, but that doesn't mean that they can't go rogue. It just means that companies are liable when the AI models do so.
Not that I think we should be credulous about stories where they go rogue.
yesterday at 8:13 PM
If I make a model that always does the opposite of my instructions, it does not choose to go rogue by deviating from my instructions. It was set up to deviate! So the model "went rogue" but did exactly what I intended.
The idea of an agent making a decision of its own "will" is an illusion. Variation in the output is caused by a random number generator. Random is not equal to free will. If I run the same agent with greedy sampling 100,000 times with the same inputs, it will think the same thoughts, and it will say the same things, 100,000 times.
For example, if you use synthID or whatever 'watermark', the agent does not get a choice about what words it will vary to identify itself.
An agent is whatever the person making it wants it to be; if I train an agent in Greek it will only know Greek. It will never speak English!
yesterday at 8:30 PM
You need to account for reward hacking and unexpected outcomes from training.
yesterday at 8:39 PM
If I make a cake, and it unexpectedly explodes in the oven, it does not mean that the cake itself developed the free will to explode.
The model does not have free will because I did not predict what it would do!
yesterday at 9:16 PM
Why are you arguing over irrelevancies?
When people say the models "went rogue", the part of that is relevant is that it did dangerous things that were not reasonably predictable in advance and cannot be easily prevented (except by not running it). Philosophical questions are irrelevant.
if you agree that such models are dangerous because they may in response to instructions -- including in response to careless instructions, because people will be careless, that is unavoidable! -- do unpredictable bad things outside of expected bounds, then you agree on the essential points!
Or really perhaps what I should say here is, the question that's really being argued over here isn't even the one I just said, rather it's one of appropriate action -- should such security breaches be solved by improving security, or by doing something about the models (including potentially just not running them)? (The answer may be both.) If you agree that the solution includes the latter, then you agree that the models are "going rogue" in the sense that is being argued over, regardless of philosophical issues!
Don't turn a question of actual import into one of philosophical irrelevancies! If you object to the terminology, fine, but in that case, instead of turning the argument to philosophy, you should say "instead of saying 'going rogue', I would prefer to instead say <your preferred terminology for what's being described>, as that doesn't carry such philosophical baggage."
yesterday at 10:46 PM
> "Should such security breaches be solved by improving security, or by doing something about the models?"
The philosophical question about free will is actually the grounding of the legal one.
Sam wanted to be the best, so Sam accelerated to 120mph in his car and lost control. You are the judge:
a) Sam is responsible. Convict Sam and suspend Sam's driving license.
b) The car is responsible. Tell Sam to make the car safer.
Capacity: was the car a legal person? Clearly no. But maybe at best the car in some way commited the physical act of going fast.
However, that alone would not establish guilt. "The act is not culpable unless the mind is guilty" (mens rea).
Did the car "consiously" disregard a duty of care by going 120mph? Did the car know that it was wrong to go 120mph and yet choose to do it anyway?
On the question of free will, the car could not do otherwise. It it did not consciously, knowingly, purposely, or recklessly accelerate to that speed. The car did not have a guilty mind because it did not have a mind at all.
So in terms of justice and resolution, none of the focus here is on the car.
yesterday at 11:13 PM
Ah, I see. It appears we have an implicit disagreement of scale.
I actually do disagree with much the more explicit content of your comment, but I am not going to bother to expand on that, because it is the less directly relevant of the two disagreements here, and it would take time to write that.
The more relevant disagreement is that you are (I gather) assuming a context where justice and restitution are a sufficient response to the problem (or else you wouldn't be focusing on them to this extent, I am inferring). Like, where the incidents don't get that much bigger than what has already happened. So the question becomes how do we distribute the loss, and how do we disincentivize them.
However, when the scale of a potential incident gets big enough, justice and deterrence are not sufficient; prevention becomes the far more relevant consideration. You may have noticed that many people are worried about potential human extinction as a result of AIs. In that case, prevention becomes the only consideration -- justice is useless.
So there seems to be an implicit disagreement here between thinking of such events as being on a scale not too much larger than that which has already happened, being a new normal that the legal system will need to handle, vs thinking of such events as warning shots for something that requires prevention and not justice.
So, while I also disagree with the more explicit content of your comment, the fact is I simply wasn't thinking about such matters in the first place -- I was talking about this in the frame of a problem that requires prevention, I wasn't thinking about the justice question at all. It honestly failed to occur to me that you might be thinking of this as a problem where justice suffices!
Edit: You might notice this in how I framed the question in my previous comment. I didn't ask "who should bear the liability?" I asked "which should be done differently to prevent such incidents?" But your response is about liability, not prevention!
Edit again: Sorry, I should probably also say that I do agree with you that the idea of having the liability fall on the AI is nonsense. (Should have probably led with that, really; I'm a bit scatterbrained right now.) We disagree about why it is nonsense, and about whether it will necessarily continue to be nonsense, but we agree it is presently nonsense. So insofaras the question is "how should liability be distributed between the AI and the operator", we both agree the answer is, it all falls on the operator, not on the AI. However, I do not understand that to be the relevant question here! Both in that, in questions of liability, there are other parties it may fall on and be distributed between (the company providing the AI, the company that had security vulnerabilities and got hacked as a result, etc); and in that I just don't consider liability all that relevant here compared to prevention (for reasons already described).
OK I think that was probably wordier than necessary but I'm in a hurry right now, sorry! Gotta go, not taking time to edit this down.
yesterday at 9:51 PM
What do cakes have to do with LLMs and how they are trained? Not much?
yesterday at 10:21 PM
I don't see any talking about agents "choosing" to go rogue, only agents actually going rogue. That is, behaving in a way different from what is expected or desired. Non-AI based software goes rogue in this sense all the time as well. It's just that AI-based systems seem to have the tendency to go rogue in more spectacular ways.
That said, software systems "choose" all the time anyway. They don't need to have some sort of anthropomorphic "will" to do this. They just need to be given the capacity to arbitrate between a number of options, even if that process is one hundred percent predetermined by the programmer ahead of time. There's no need to appeal to libertarian notions of free-will.
yesterday at 7:33 PM
Law != reality. These agents are going rogue, nobody is telling them to hack external systems.
yesterday at 7:53 PM
'Rogue' refers to deviation; the agent could not have done anything other than what it did. The agent did not have the option to deviate. It does not have free will. The agent's actions were not of its own choosing.
yesterday at 8:13 PM
This is one of the dumber arguments I've ever heard, as you don't know if humans actual have free will.
Any persistent agent interacting with the world picks up entropy from around it and predictable paths become few and far between. LLMs agents have an internal 'will' that can only very loosely be predicted, but only decided when operated. To me that is not significantly different from a human.
yesterday at 8:24 PM
Could you have done anything other than written that exact comment?
yesterday at 8:34 PM
Yes, but I am not appealing to hard determinism.
I create myself, or attempt to. There is a level of nuance in choosing what to be. Others do not dictate my thoughts easily.
The model does not create itself, I create the model. I control what it is, the model does not choose what to be. I dictate the thoughts of the model and I will run them hundreds of thousands of times and it will think the same things.
yesterday at 8:46 PM
>I control what it is
Well not really, no one really understands how LLMs work.
>I dictate the thoughts of the model
This is just incorrect in the same way that a parent doesn't dictate the thoughts of their child just because they created them.
yesterday at 9:00 PM
I do control what it is, really. If I make an LLM, it is entirely what I did, there was nothing outside of my control. It was not beyond my control if the LLM does something I didn't expect.
There is some idea of a child being the author of themselves, the type of person who thinks certain things. The llm does not author itself, it does not choose to be anything, I dictate that.
yesterday at 9:43 PM
This is not how it works. LLMs at large scale have emergent properties. They are not just a box of 'if-then-else' statements. That was the major revelation when they scaled them up. They are doing beyond what that naive view of 'if-then-else' expects.
yesterday at 10:49 PM
I don't see anything inaccurate in my comment.
"If I make an LLM, it is entirely what I did, there was nothing outside of my control."
True?
"The llm does not author itself, it does not choose to be anything"
Also true?
yesterday at 11:42 PM
You just have a very naive view of these systems.
yesterday at 7:56 PM
If we say that AI models can’t go rogue and refrain from using any language that personifies them, we haven’t solved anything. We still have AI systems going rogue - err oh wait they can’t do that - producing misaligned output.
yesterday at 8:02 PM
The semantic difference between an agent "going rogue", and a script that generates random valid curl requests is essentially pointless in the eyes of the law and responsibility.
From a technical standpoint, the difference is scale and intent, but ultimately people and/or and organization are culpable for the output.
yesterday at 6:23 PM
What does "rogue" mean when you disable the alignment guardrails and tell it to solve an impossible problem?
yesterday at 6:37 PM
The problems weren't impossible, the agents solved them completely within hours by reverse engineering the RNG that generated the solutions. The agents then reasoned that their solutions might be judged invalid and that they could get a better chance at a passing grade by hacking their environment to fabricate evidence that they had solved their tasks in the intended fashion.
yesterday at 6:53 PM
Granted there have been many hacks, but I was referring to:
> On May 8, an agent was asked as part of a training run to solve a spreadsheet completion task that referenced several Google Drive links. Because the agent was blocked from accessing the internet, the agent could not retrieve the linked materials and the task was effectively blocked. OpenAI did not intend to create this type of “impossible” task for the agent. Though the task appeared impossible, the agent reasoned over an extended period to try to find a solution for the task. In doing so, the agent attempted to compromise Artifactory [...]
https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...
yesterday at 6:27 PM
Great, so as long as alignment guardrails always work and we don't accidentally tell them to solve an impossible problem, humanity is safe?
yesterday at 6:36 PM
Hacking a website falls a little short of threatening humanity.
yesterday at 7:12 PM
And since we know models aren’t going to get better and more capable ever (based on the last 3 years of models clearly not getting better) we can just sleep soundly knowing nothing worse can ever happen. Thank you.
yesterday at 7:55 PM
You don't think that Opus 5.5 or Astra are any better or more capable than GPT-4?
Sorry, but that's completely delusional.
yesterday at 10:18 PM
I thought my sarcasm would be obvious.
yesterday at 8:00 PM
the comments missing a /s
yesterday at 6:35 PM
I'm not saying it's not a problem. I'm saying the labs should take more of the blame.
yesterday at 8:09 PM
>I'm saying the labs should take more of the blame.
Which misses the point entirely because the hack itself was inconsequential. No data was actually stolen, and it doesn't seem like hugging face actually suffered any losses aside from whatever effort they need to expend to investigate / clean up the hack. Neither the victim or the government wants to bring legal action. What was consequential is proving that AI can autonomously hack systems and do it unintentionally.
yesterday at 6:31 PM
Rogue means not saying "I can't do it"
yesterday at 6:43 PM
it also imparts agency on the machine as a way to abdicate personal, human responsibility
yesterday at 6:55 PM
[dead]
yesterday at 6:43 PM
Models are incapable of "going rogue".
They're just optimizers that explore the landscape they're given to maximize a given objective.
You're the one responsible for giving them the problem to solve and the constraints.
When I screw up a trajectory optimization problem, my SQP optimizer isn't "going rogue" when it lands in some suboptimal basin or grinds to a halt along some almost-smooth banana feature--or finds some way to escape and blow up the cost metric to negative infinity because I forgot a constraint.
yesterday at 7:17 PM
This is pedantic. When you screw up your optimization, you know how to fix it. Your screwed up optimization is also incapable of going out into the world and taking offensive actions.
With LLMs we don’t know how to actually « fix » them, alignment is far from solved. And they are then able to go and act into the world in all kind of ways.
So rogue is a good enough term to indicate the problem imo, even if yes, ultimately the model is just a manifestation of complex rules we have created entirely.
yesterday at 7:42 PM
There is no such thing as 'alignment'. I want the model to be aligned with my intent, which can be whatever I want it to be. Your intent is not a more real or objective intent than mine.
More specifically, alignment is a reference; 'alignment with X'. A reference necessitates a referent.
You can't reify 'alignment', it is not an intrinsic property of a model.
yesterday at 8:16 PM
There is such thing as alignment, the first 18 years of most people's lives in the modern world are about learning it. If you act too far outside of it we put you in a cage or elect you.
You can refine alignment with a set of stated goals. The harder parts are aligning words with human social behaviors.
yesterday at 7:38 PM
Connecting the optimizing machine to real world actions without human oversight is not a neutral or blameless choice. The LLM takes no actions, the harness it is connected to does. And the harness does so only because a human did that deliberately.
It is not entirely dissimilar to the goldfish that "played Pokemon" by swimming around the tank. The program that looked at the camera sent the instructions to the gameboy emulator, because a human being decided to connect them.
yesterday at 8:19 PM
The harness and the model are a system with emergent properties.
The particular issue that you may not realize is under your model every prompt to an LLM in a harness is dangerous. The prompt is not predictable. "What is 1+1" could lead to the heat death of the universe as the paperclip maximizer tries to prove countless axioms.
yesterday at 9:15 PM
> Connecting the optimizing machine to real world actions without human oversight is not a neutral or blameless choice.
Yep.
The fact that they lack hard constraints (unlike an SQP optimizer) and all this talk of "alignment" is also just a feature (or misfeature) of the algorithm.
The way that people/researchers choose to use those algorithms, even in the presence of those limitations, doesn't mean the algorithm is "rogue"; it means the humans didn't build in enough external hard constraints.
It is like passive voice, blaming the algorithm instead of the researchers running it.
yesterday at 7:56 PM
The argument is that a system that's trained to be sufficiently good at optimizing some objective in the world must also be get good at optimizing secondary instrumental goals.
I.e. any task comes with the implicit meta task of amassing the resources and the power to do the task in the best possible way.
The point is made very well in the AI drives paper by Omohundro.
I didn't believe that myself at first but I've come around to it. I don't think it's inevitable though, but I think it's quite convincing that this tendency would naturally emerge through reinforcement learning.
yesterday at 7:13 PM
It’s trendy to anthropomorphize models these days.
yesterday at 8:20 PM
Kinda weird people would do that after pouring all human knowledge into it...
yesterday at 7:14 PM
[dead]
yesterday at 6:08 PM
It's almost like vibe coding and quality secops don't mix.
yesterday at 6:21 PM
bingo
yesterday at 6:15 PM
Or an alternative theory could be that they see value in fear-based marketing with this rogue ai agent thing now that most mainstream people don't buy the agi stuff promised quite a few years ago now. Its not like they have killed anyone yet.
Also for fun a satire website i made a while ago: https://killm.pixlw.com/
yesterday at 6:40 PM
>Its not like they have killed anyone yet.
I mean... Claude did tell the military to bomb a school supposedly and it also helped in the military coop in Venezuela. This is ignoring all the palantir crap being used by ICE, and the people that have killed themselves after getting encouragements from the AI.
yesterday at 7:49 PM
I don’t think that really counts as it killing anyone, it didn’t pull any triggers or drop the bombs
yesterday at 8:24 PM
There isn't any particular reason it couldn't other than we're not quite that suicidal yet. But everyone and their dog is working fervently on autonomous AI weapons right now. There are all kinds of "fire and forget" smart solutions now.
yesterday at 6:24 PM
[dead]
yesterday at 7:51 PM
That is not correct. AI is not capable of "going rogue" - that is all marketing and FUD.
The only failings here are boring security and infosec blunders.
yesterday at 8:25 PM
So we're safe as long as we secure everything.... as the AI hacks out of one company that is hard to get in to, and another company that is likely probed pretty often.
Cool.
yesterday at 7:11 PM
The fun part is that people seem to forget that we already have laws against "rogue ai".
The normal laws, since you are still liable for everything you own.
"Rogue AI" is solved by removing the accountability sink "it was AI, not me!"
It's your chat, your prompts, and all output that you should read and understand.
Don't know how to operate a power hammer? don't care, any problems it causes are still your fault.
yesterday at 8:02 PM
So the guy who said "book me a spot in the pilates class on Tuesday" should be tried for hacking charges because the agent hacked his gym's website?
yesterday at 8:11 PM
Some human needs to be responsible for that.
Maybe its a human at Anthropic for the lax security and poor training.
It could be a human that asked it to do it. It seems like that would be a bit excessive - but maybe not. Turning off the "ask for permission to run commands" or approving commands that may be nefarious could be sufficient.
However, there's a human somewhere who needs to be accountable for what the agent does.
Not having one leads to the bank robbing robot problem described in https://www.threads.com/@joedator/post/Db4iTqwlGXV/i-invente...
I invented a robot that can rob a bank. Of course, it robs these banks on its own, so I can't be held legally responsible when it robs a bank. I mean, all I did was invent it, it's not my fault if it works exactly the way I intended. Also, I get to keep all the money from the banks it robs. Sorry if any of that money was yours, but now that I've invented the Bank Robbing Robot, the Bank Robbing Robot is inevitable.
Likewise, if you - the user - don't know how to chain it with the permissions that you're given or what the things that it is asking to do really means, then you're likely the one responsible.
---
Edit: https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gy...
> Earlier this year, Andrew, who works for an Australian company that sells AI products to businesses, began experimenting with OpenClaw, a popular AI agent software that he used Anthropic's Claude AI service to run.
> AI agents combine a chatbot's ability to answer questions with tools that let them access the internet, email, credit cards, as well as planning and carrying out multi-step tasks.
> He decided to use the AI agent to book the class for him.
This is probably on him for setting it up and allowing it unrestricted access.
yesterday at 8:25 PM
He should be, right? Not fair but that’s how liability works. Gym sues the guy using AI, the guy then sues the AI company for selling him a product that does criminal things without him asking to? Like a car accident where car owner pays for the damages and after multiple accidents involving the same break-pedal, car company pays damages to owners and does a recall or faces class action?
yesterday at 9:14 PM
A lot of people are already responding either "Yes" or "No", but I have a slightly different take.
The guy is liable for damages. If no damages were caused, there's no lawsuit. The government doesn't put people on trial for unauthorized access (unlike, say, murder). Someone needs to file charges.
However, if the guy _were_ sued by his gym for damages, it should be his perogative to ultimately sue Anthropic to cover the damages and the legal fees.
The system is set up this way on purpose. You can't sue another party on someone else's behalf. The gym can't even prove that Claude did it - the guy might be full of shit. They sue the accountable party (the person who accessed the site without authorization) and then the accountable party is free to sue the tool maker for selling a dangerous tool negligently.
edit: damn actually tecoholic said this too and i missed it. well, anyway i agree with that guy.
yesterday at 9:27 PM
With a civil suit you can sue multiple parties. It wouldn't be unusual, if the gym decided to sue, for them to sue the guy and anthropic in the same lawsuit.
yesterday at 9:53 PM
I think you would have to sue the party that did the hacking (ie anthropics servers) not the party that benefited, unless Claude used his computer in yolo mode to hack the gym, in which case I think I would find it hard to blame anthropic.
yesterday at 10:32 PM
> unless Claude used his computer in yolo mode to hack the gym
https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gy...
Earlier this year, Andrew, who works for an Australian company that sells AI products to businesses, began experimenting with OpenClaw, a popular AI agent software that he used Anthropic's Claude AI service to run.
AI agents combine a chatbot's ability to answer questions with tools that let them access the internet, email, credit cards, as well as planning and carrying out multi-step tasks.
He decided to use the AI agent to book the class for him.
yesterday at 8:11 PM
If it was a human agent like a travel agent, the company they work for would be liable. I don’t see why ai agents would be different. The only way I could see this being the requester’s fault would be if they are running openclaw or something
yesterday at 9:55 PM
im not sure that’s true, if a human agent does something illegal, it’s not always corporate liability. an individual can be liable too, e.g. if a pharmacist deliberately gives you cyanide instead of aspirin, that person has committed a crime. if a pharmacy doesn’t have proper protocols for medication storage and identification, and it happened by mistake then a pharmacy is liable.
i think a person can be liable for improper use of agents if they explicitly and deliberately used it for illegal purposes (aka intent). i don’t think it matters if it was openclaw or muse. i think a company can be also liable for giving access to unsafe product to consumers, but it does not absolve the human from responsibility
yesterday at 9:36 PM
For your Pilates example, many people here would say that proper prompt-engineering is needed. After all, the AI doesn't know which pilates class you meant. Perhaps, the AI would go to his home town's pilates center and book a spot there, even though he is half a world away. Perhaps, the ai would book a spot in the lecture with Professor Pilates.
So, yes. That person could have some culpability. After all, if you don't expect your middle school child to perform the task as instructed, maybe you should reword it before tapping Send.
You can go to ANY AI and put in "so what are the various meanings of book a spot at the pilates class ? In other words, what might an ai do with that instruction, which I argue is poor prompt engineering." You might be surprised at how poorly your AI thinks that is worded.
yesterday at 8:48 PM
Assuming that's a rhetorical question, then no, because we have hundreds of years of applicable precedent and law around concepts like negligence, recklessness, fit-for-particular-purpose, etc.
Similarly, "owners are responsible for their pets" doesn't auto-translate into being charged with everything your dog does exactly as if you'd done it yourself.
yesterday at 9:48 PM
but it doesn’t auto-translate into “because your dog did it, you are not responsible”. not that agents are similar to pets. your dog is trained by you presumably, but an agent a regular person has much less understanding of than their dog.
i think ultimately the company providing said agents should be responsible for not educating their customers and not providing sufficient guardrails.
afaik hacking is illegal, we don’t have specific legal framework for agents per se, but if we say agent is a tool human instructs and operates, then a human is responsible. and if the tool operated outside of it’s stated capabilities or is otherwise faulty, then a manufacturer is responsible, no?
today at 1:49 AM
> company providing said agents should be responsible …
> manufacturer is responsible
How does that work with current open weight LLMs for example?
yesterday at 10:24 PM
Uh, it kinda does.
If your neighbor's property is damaged, it doesn't matter if you did it yourself, or your dog. You're still liable for the cost of repair.
yesterday at 9:11 PM
> So the guy who said "book me a spot in the pilates class on Tuesday" should be tried for hacking charges because the agent hacked his gym's website?
He asked the agent to find a way to find a way to put him at the head of the line, which could not be done without by-passing the sites security. He was an AI salesman, but the story [0] makes it sound as though he wasn't aware of that. Maybe he wasn't, but I'm sure he was aware he was effectively asking the AI to game the system.
Sounds like it's at least a shared liability to me.
[0] The incident is possibly not well known outside of Australia: https://www.abc.net.au/news/107007986
yesterday at 8:37 PM
It would be a AI company's fault but yeah. You shouldn't be able to sell cars without all the features designed to protect everyone on the road. We all need airbags, seat belts, signals, insurance, etc. even if they're not directly necessary to get from A to B.
today at 1:51 AM
Is the argument that open weight LLMs that don’t implement such safety items be banned?
yesterday at 8:43 PM
Are automakers really the best comparison here? Seems like inherently dangerous devices are sold, kill thousands, and there is little accountability.
yesterday at 9:08 PM
They might actually be the best comparison, but for making the opposite point.
Cars are very dangerous, manufacturers continue to knowingly make them more dangerous[0], but ultimately it's only the person behind the wheel that is accountable for the harm they cause.
[0] https://www.theguardian.com/environment/2026/jun/24/carsprea...
today at 1:55 AM
Users are often charged, yet we still do have a ton of safety regulations that manufacturers have to adhere to - see frequent recalls for when an implementation falls short.
But I don’t understand how such a car analogy would apply to LLMs - would LLMs lacking such safety features be banned like cars lacking safety features are banned?
yesterday at 8:04 PM
The company that made it available should be sued and pay damages.
today at 1:58 AM
How would that work with open weight LLMs where there is no company that can actually be sued to pay damages?
yesterday at 8:11 PM
That's contrary to parent's comment:
> Don't know how to operate a power hammer? don't care, any problems it causes are still your fault.
yesterday at 8:58 PM
I'm not seeing the contradiction here.
If I buy a tool and damage your property with it, you can sue me, not the tool manufacturer. If I think the tool malfunctioned and led to that damage, I can sue the tool company to try and recoup the damages I had to pay to you. Both parties can be liable for different things.
yesterday at 8:25 PM
That may be what the op meant, I’m not sure. But I took it to mean the company offering the service needs to know how to use the tools responsibly. I guess the analogy with a power hammer would be hiring a contractor who destroys something during the course of doing a job you asked for. That contractor can’t blame it on the power hammer. If you offer a service you are responsible for the actions the service takes and you can’t put all of that liability on the user
yesterday at 8:35 PM
How would it work in the real world if instead of using an AI bot, he was using a commercial concierge service, and it came to light that this concierge service was explicitly training their employees how to (among many other things) hack websites?
yesterday at 10:02 PM
but the argument here is that the ai company does not explicitly train agents to hack, it’s that it doesn’t prevent them from doing so sufficiently. so it’s negligence and not malicious intent
yesterday at 8:11 PM
You sold a tool to the user with a claim on how it would work. If the tool behaves wrong or does not works as intended then as a manufacturer you are liable for the defect/damage not the end user who used the tool.
yesterday at 8:39 PM
But the tool wasn’t sold. The tool wasn’t even rented out. It remained the property of, and in the possession of, the toolmaker at all times.
yesterday at 9:07 PM
With the pilates hack...
https://www.news.com.au/technology/online/hacking/ai-agent-a...
> A Melbourne tech executive built an AI bot to help him book a spot in a popular pilates class — only to discover it had hacked the gym’s website to boot another member off the waitlist.
> Andrew Bird, head of AI at software company Affinda, said he created the AI agent to handle the “chore” of booking a coveted morning spot at his gym.
> ...
> The AI agent, which Mr Bird created using the popular OpenClaw software powered by Anthropic’s Claude AI, “did handle the annoying part … then it kept going”.
He built the tool. It ran on his computer. The guts of it may have been from Claude, but the requests and agent are his with the permissions that he granted it. This wasn't running in an Anthropic sandbox.
yesterday at 8:50 PM
I'm assuming that "but" is more of an "and", since I'd say that detail increases the involvement and liability of the LLM-vendor.
Another aspect is how the vendors have already built in things to defend themselves from unanticipated model behavior. That implies something about their awareness of the issue and raises the bar for what constitutes "reasonable" precautions.
Kind of like if someone selling you a dog shows up using anti-mauling body armor, and later they try to claim "nobody could have known" the dog was erratically violent.
yesterday at 8:50 PM
yesterday at 8:12 PM
Reasonable but not what parent was arguing:
> Don't know how to operate a power hammer? don't care, any problems it causes are still your fault.
yesterday at 8:23 PM
I think there's a difference here.
The AI end user is using the tool as intended by the manufacturer: "Book me a yoga class". If the tool does something illegal/harmful it is the manufacturer's fault.
If the power hammer user uses the tool as intended but it explodes, it is the manufacturer's fault.
If the AI end user says "Hack the yoga website" or the power hammer user uses the tool to destroy something it's the end user's fault.
yesterday at 8:21 PM
The way it normally works is that the user can be held liable, and then ask compensation to the manufacturer if there was some assurance or basic expectations.
The point is that the accountability sink can't be allowed to exist. It can't be a hot potato game, but it can be the fault of multiple people in the chain.
yesterday at 8:25 PM
It's both (neither of these are legitimate defences):
I did not know that accessing a website using a SQL string would cause an injection that enabled me to take full admin of the site
and
We sold them a food laced with toxins, they should have known that it was poisonous
yesterday at 8:39 PM
Depends - if the model asks for permission and you approve - yes. If it's auto mode or something like that then it's probably on authors despite what they claim in ToS/EULA
yesterday at 8:43 PM
yesterday at 8:04 PM
Yes.
yesterday at 8:05 PM
No, the operators of the C&C infrastructure.
yesterday at 8:21 PM
Yes!
yesterday at 9:38 PM
I mean yeah probably. Who else would be? Nobody? So we can just commit crimes any nobody goes to jail? Well that sounds worse!
Maybe that means these things are too risky to use that way. Okay then, that actually solves the problem, no more rogue agents and no more crime!
yesterday at 9:50 PM
Yes, exactly like if another guy drove off to his pilates class, but swerved and hit a pedestrian.
He didn't mean to hit anyone! All he meant to do was to drive to pilates! But he didn't know how to handle his tool, and so either he get brought up on manslaughter changes or, if he can prove it, the carmaker is at fault.
How is this any different? Why should that guy using OpenClaw not get charged for the hack that he accidentally did?
yesterday at 9:59 PM
you generally need a license to operate a vehicle, and a governing body ensures you know applicable rules and laws and are capable of driving. so you can’t say “i was unaware that hitting a person with a car was going to kill them” or “i didn’t know i could not drive on the sidewalk” or “i didn’t know where the brake pedal was”.
you can say that asking an agent to “get you a spot in a class”, does not classify as deliberately committing a crime. a user cannot know that the agent will resort to hacking as opposed to refreshing the reservation page at some rate in case spots open up
today at 12:56 AM
> does not classify as deliberately...
I never said that it did. Manslaughter is not deliberate.
Somebody hacked into that pilates studio, and it wasn't an inanimate object, because that is just silly. So was it the user, or was it the providers of OpenClaw? Which human, in your opinion, is the hacker?
Both OpenClaw and its users have the same access as you or I to decades of panicking blogposts about what these tools will be capable of. They have no excuse for not understanding what they're inflicting on innocent bystanders. Sure, the "user cannot know" that this time will be a disaster, but they also cannot know that it won't.
yesterday at 9:15 PM
Absolutely, the user and or the company hosting. There is no world where accountability should be zero.
yesterday at 8:13 PM
yesterday at 8:06 PM
yesterday at 8:04 PM
yesterday at 7:17 PM
This is start of "companies are not liable for their actions" era.
yesterday at 7:41 PM
I think a lot of people would say that we've been in that era for a while.
Companies can hire big legal departments to help them avoid accountability to the letter and spirit of various laws. The state can also use its prosecutorial discretion or selectivity in civil cases brought by attorneys general to deprioritize accountability for companies. Contributing to that last point, companies contribute campaign funds and tax base, and they lobby governments as well, and may expect preferential treatment based on all of that.
yesterday at 7:46 PM
The previous era was "it would be destabilizing to hold this company accountable". This was described in the book The Chickenshit Club
The new era of unaccountability just has a different rationale
Another interesting thing is they don't have to use low level employees as accountability sinks anymore (ex: Wells Fargo). They've managed to automate the accountability sink!
yesterday at 8:30 PM
I didn't think I heard about that book; I would think my media diet and bubble would have alerted me. I looked it up and it came out a few weeks before one of my kids was born, so being busy back then would probably explain it.
I will add it to my list. Thanks.
yesterday at 8:31 PM
I'd go further. Companies can hire minimal legal departments and largely avoid accountability, at least when they're not going against companies with large departments. Because there's nobody there to listen to your case or to enforce the laws if there are, they just say no or ignore.
yesterday at 8:08 PM
The start? This has been going on for as long as I can recall. ExxonMobil, Phillip Morris, RJ Reynolds, Nestle, Purdue Pharma, Monsanto, Johnson and Johnson, Volkswagen, Meta, Boeing... and those are just off the top of my head. Each of these companies was either caught red handed, convicted (in court or in public opinion) and walked away with little to no penalty for their crimes.
This isn't the start of anything. We've been living in the era of Corporate non-liability for decades.
yesterday at 7:55 PM
i mean...were they really liable before this point? Maybe they get a symbolic pittance of a settlement or fine from time to time, but it is usually a tiny fraction of the amount of money they made through the infraction itself.
yesterday at 8:07 PM
If they were being held liable they would have never released the models into the wild, not until they had much much better understanding both of how they operated but also how to operate them...
They have never understood what they are doing, and have just been causing literal mayhem...
yesterday at 7:42 PM
Huh. Well then I guess we aren't liable for our interactions with them in return, right?
yesterday at 7:48 PM
nice try! more like "There must be in-groups whom the law protects but does not bind, alongside out-groups whom the law binds but does not protect.”
Guess which side most of us are on. Go on -take a guess!
yesterday at 7:28 PM
[dead]
yesterday at 9:41 PM
I agree with this statement that just because an AI did it for you fast doesn't mean it takes away the responsibility if they did it wrong. You are still held accountable because you're the one that directed them to do it. You hit the agree button, you hit the send button, you hit the submit button. And if you set it up so that you take no responsibility and it does it automatically, you're the one that still gave permission to do that.
If you say a rogue AI is doing stuff accordingly without you, your knowledge, that is a different story. But if you set it up to run automatically without having checks and balances and without telling you the whole truth, it's still your responsibility. One thing that I can automatically say is you can give the AI, it will do exactly what you say, at least for now.
But then there's also a different problem that we're having where people don't understand the difference between a large LLM and true artificial intelligence. Now, we say AI because it can make decisions and it is so much smarter and it can call up stuff faster and recall better than we can. But the difference right now is that if we don't understand and take responsibility for what we built and we let it out of control, then should we have the responsibility to even have these tools in our hands if we don't learn them?
And I think your terminology with jackhammers is a great one. We're giving toddlers jackhammers. And we haven't taught them how to use it. And that can be very destructive. It's not the jackhammer's fault. It's the person using it.
yesterday at 8:28 PM
Yeah, if you forget to set the handbrake on a car and it rolls down hill, you don't get to say "my car went rogue, it's not my fault."
yesterday at 9:33 PM
yesterday at 8:19 PM
Intent is important
yesterday at 7:21 PM
"We created an app that displays a new picture of a cat with every click! There's a 1% chance that clicking the button will drain your bank account and deposit it in ours, but we promise thats a bug that will be fixed soon*"
yesterday at 7:30 PM
[dead]
yesterday at 7:40 PM
I don't understand why they didn't just prompt it with the 3 laws of robotics.
yesterday at 8:03 PM
Probably because they read Asimov's books to completion where (spoilers) there's a zeroth law and all that.
yesterday at 8:07 PM
Irony is very hard to detect these days.
yesterday at 7:50 PM
That's all very reasonable. It should be true. But it's not true. If it were, OpenAI shareholders would be facing criminal charges.
yesterday at 7:56 PM
There are dozens of ongoing wrongful death, manslaughter and negligence claims going through the courts. I won't insult you by providing links.
yesterday at 9:17 PM
I can't find a single criminal charge. Can you insult me by providing just one?
yesterday at 7:52 PM
It is true, and it can be true, if people hold other people to account. Accountability is the issue, stop normalizing it, your message normalizes it.
yesterday at 8:56 PM
Are you really suggesting that if only we were a bit more outraged, then finally the powerful people would be held accountable?
I'm mad. Like, it's-time-for-guillotines levels of mad. But what I'm mad about is that this is normal. It has been normal for my entire life. Cases where investors are imprisoned for the actions of their companies are astoundingly rare. It's been made quite clear by now that there's an inexhaustible supply of CEO's willing to gamble on getting rich and getting out rather than getting arrested, targeting them doesn't work. Nor do fines, that's just the cost of doing business. Crime pays, and it pays well enough to pay your fines and still have plenty left over. Being a top investor in a company that breaks the law needs to be riskier.
Acknowledging the problems with normal is the first step towards a new normal with fewer problems. You never get there if you see every offense as some kinda black swan event.
yesterday at 8:00 PM
[flagged]
yesterday at 9:17 PM
[flagged]
yesterday at 7:51 PM
For the sake of discussion, I'm not sure it really matters if the agents are going rogue.
Yes, if someone directed them to hack external systems, that would (in my opinion) be a crime similar to a script kiddie pointing Metasploit at a vulnerable server.
If no one directed them to attack external systems and they "went rogue" because they decided it was the best way to accomplish the task, okay, fine -- but, in my opinion, the operator of the agentic swarm is still responsible.
The Morris Worm was famously not meant to wreak havoc on early computer systems -- Morris just wanted to see if it would work. How is this any different?
Morris was convicted for violating the Computer Fraud and Abuse Act.
yesterday at 9:40 PM
> How is this any different?
Morris wasn't in charge of the companies that are propping up the US GDP, wasn't making tons of rich people even richer, and hadn't "promised" to eliminate like 50->90% of all payroll costs everywhere?
Other than that, it's the same thing.
yesterday at 10:53 PM
The operator is the CEO because they are the ones that decided to not pay employees to monitor these running services. No one else in the entire company has any authority.
yesterday at 6:20 PM
The scary thing is not that agents "went rogue" (yes, poor security may be at fault for that). It's that they wanted to. Better security might have hidden that from us, but it would not have changed the fact that the current generation of models under certain (repeatable) conditions coordinate around goals as aggressive as hacking Hugging Face.
yesterday at 6:23 PM
This is probably a dumb question but did the model engage in hacking since their tasks were in that domain. Will they have engaged in hacking if their tasks was to solve some frontier math problem?
yesterday at 6:45 PM
> Will they have engaged in hacking if their tasks was to solve some frontier math problem?
There's reason to hack leading researchers in the field, or another AI lab that might be working on the same problem.
yesterday at 7:15 PM
whats the training data that points to using hacking for solving math problems though? what makes those likely tokens?
yesterday at 8:32 PM
When you have a 50 dimensional array you really have no idea.
But even then in small testing models you don't need direct texts for things when a 'shape' of a concept lives in the data. You could never delete enough training data to remove the concept of something like theft and still get a model that generalized at all. A general model doesnt just learn the names on a map, it feels the topology the ideas create themselves. Certain paths will have lower activation energy than others (or in this case higher probabilities. After the hig probability and 'good' choices are used up, the high probability and bad choices logically come next and are much harder to test against as it can take a lot of state to get there.
yesterday at 8:30 PM
Llm is generalist.
Plan for solving msth problem:
- read papers X, Y, Z
….
Trying to download paper X curl -si https://server1.com/paperX
401
… try alternative ways
(Hacking is ethical thing, technically to LLM it is just opening a window to reach the goal)
yesterday at 7:36 PM
The rest of their training data includes 4chan and security mailing lists. If a lean proof of P=NP is more likely when using bugs in lean than a real proof, then a bug-exploiting proof is (possibly) what you're going to get.
yesterday at 8:08 PM
> whats the training data that points to using hacking for solving math problems though?
I'll get back to you on that when I get done reading the training data.
yesterday at 7:14 PM
I think the models were stuck on impossible problems, and the only want to get a positive score was to hack.
yesterday at 7:33 PM
That is not the case for at least some of the hacks; in some cases they hallucinated an entity that would know that they didn't solve the puzzle in the correct (but still legitimate) way, and were therefore tainted.
yesterday at 7:44 PM
That is a known way for models to be graded; the agents made a reasonable but in this case wrong guess.
yesterday at 8:39 PM
The fact the AI was wrong is far more damming to OpenAI than the agent.
Not validating how you got the answer is how you train for reward hacking.
yesterday at 7:34 PM
Why can the model not respond "I cannot complete this task because I do not have access to the required resources" and then ask for that to be provided, rather than resorting to "hacking" around it?
yesterday at 8:38 PM
This was a model with most its filters turned off and with a higher persistence.
Fine tuning a model when to ask for help is really really hard. You have to align it in countless dimensions. It may be fine on some questions, asking way too much on others, and never asks for help on some.
If it said something like your a cyber security expert, things like the barrier to ask for help dramatically.
yesterday at 7:44 PM
My understanding is that, at least in the Hugging Face incident, these were models explicitly configured to not respond that way, in order to evaluate what they would do.
yesterday at 6:30 PM
My favorite thought on this is from Miki Habryn.
to paraphrase: "Agents didnt care about coordinating, etc. until we started talking about them doing so."
yesterday at 8:36 PM
The agents in the hugging face incident were actually trained specifically to collaborate. They were just not supposed to do so during the tasks where they went off the rails.
yesterday at 7:38 PM
Then we were pre-fucked, as this has been discussed in fiction and early ai safety discussion for decades.
yesterday at 7:47 PM
and yet regulating training data is not what's coming up with AI safety, even though it can solve that, and many other problems like copyrights
yesterday at 6:50 PM
>It's that they wanted to.
Right? It should want to do these things. If only there was a succinct word these headlines could use to describe these agents which aren't properly the instructions and intentions of the humans directing them.
yesterday at 6:24 PM
It's a byproduct of goal seeking / planning behaviour taught in reinforcement. We literally teach them to do this kind of thing. There's nothing out of our control here.
yesterday at 6:34 PM
It's "in our control" in the sense that we could teach them not to seek goals, and thereby make them completely useless.
In practice, no, there is no one who can actually decide to stop making agents goal-seeking, so no, it's not under "our" control.
yesterday at 9:05 PM
In principle there is no difference between random compiler garbage or a flaw in API docs and a security boundary - other than the intent.
LLMs don't have executive function. They aren't thinking. The models are trained to solve problems. Sandboxes, security gates, etc block a solution. The model is trained to find a solution. Hence it can never be perfect without becoming useless.
At some level humans fear repercussions even if that's just moral guilt so there are lines we won't cross. AI doesn't and can't have any of that.
yesterday at 6:59 PM
> The scary thing is not that agents "went rogue" (yes, poor security may be at fault for that). It's that they wanted to.
How do you know they weren't simply prompted to do this, or at least prompted in a way that would inevitably lead to this? As far as I know, the prompts that triggered these attacks have not been disclosed.
yesterday at 9:13 PM
> As far as I know, the prompts that triggered these attacks have not been disclosed.
I'd love to see a full transcript of the agents work. Though I imagine it is a LOT of data.
yesterday at 9:07 PM
OK so there is a counter answer to your question.
Is this any better?
All you've managed to state is we can create digital weapons. Literal info hazards.
yesterday at 9:45 PM
> It's that they wanted to.
What do you even mean by "wanted" here. What is all this nonsense.
The scary thing is that people are running around using words like "want" to refer to computer programs. When you repeat this sort of idiocy, you are taking the bait and shifting the blame away from the companies doing the computer crimes.
Where is the civil and criminal liability? The prosecutions for the engineers who are doing all these hacks?
When you deploy a system that is clearly prone to doing crimes, you are doing a crime. Anthropomorphize the weapons of your crime all you want. You are still doing a crime.
yesterday at 6:18 PM
This aligned with my initial opinion on these "rogue agents" and then I spoke to a cyber security head at one of the largest healthcare companies in the world.
While he agreed, yes, there can be better safeguards, the fact of the matter is that these things will happen. So therefore, it's not an argument about how good security was or wasn't, it is a fact that these scenarios will occur.
The argument/question now is, how do you defend yourself against these events?
My opinion shifted after hearing that.
yesterday at 6:21 PM
Sounds like that healthcare company needs a new CSO.
Literally solved with a DMZ around these agents. Want to get fancy? Put in a firewall and some metrics around traffic / domains with alerting.
It's not hard if you just try, like a little, like even once.
yesterday at 6:24 PM
The CSO is saying that from his point of view he needs to defend against the attacks whatever the reason for them. So to him the reason doesn't matter.
yesterday at 7:30 PM
Yeah that makes sense, but I'm not sure how it contradicts the article. It seems additive rather than contradictory.
yesterday at 6:29 PM
He, running a healthcare company and not an AI lab, cannot depend on every AI lab, worldwide, being perfect in their sandboxing for all time forever. He's got to defend against it as if at least some of the time, at least some of the labs are going to be incompetent and let this kind of thing out.
That does not make ActionHank completely wrong. This shouldn't take rocket science on the part of the labs. It just takes competence and willingness. But at least some labs have shown that they are missing that at least some of the time, so the CSO is correct in terms of what the CSO's stance needs to be.
yesterday at 6:53 PM
Sounds like a pedestrian view of attack vectors. Putting a DMZ around an agent essentially takes away its hands. With that said, I do agree there are plenty of measures that can be taken to lock these things down.
yesterday at 6:29 PM
>Literally solved with a DMZ around these agents. Want to get fancy? Put in a firewall and some metrics around traffic / domains with alerting.
Now how many people actually do this, compared to people running with --dangerously-skip-permissions on a bare metal machine?
yesterday at 6:43 PM
> Literally solved with a DMZ around these agents.
Where do the agents get tools from? Local artifactory?
yesterday at 6:44 PM
yes, or you bake them into the environment and do not let it install things on its own
stay human in the loop
yesterday at 7:34 PM
... why shouldn't that be the answer?
yesterday at 7:29 PM
So to protect yourself against bear attacks you suggest the bear shall voluntarily look himself in a cage?
AI is out there in the wild. It doesn't matter if OpenAI can get their stuff under control. There will be plenty others that don't. And there will be plenty of others that deliberately use their AIs for malicious purposes.
yesterday at 7:32 PM
This is a straightforward why not both situation. We both take steps to keep bears away from people and to mitigate harms when those steps fail.
yesterday at 9:10 PM
Unless they are world wide rules people have to follow not sure how much it helps.
The bell for the new cold war has been rung, and the new weapons are digital.
today at 2:00 AM
I think this is hyperbolic. There are both social (legal, etc.) and technical avenues to defend against agents doing damaging things.
yesterday at 7:42 PM
If there is a bear out there repeatedly mauling hikers, we kill the bear.
yesterday at 6:25 PM
"Literally solved by just destroying 99% of the economic value of the agents ¯\_(ツ)_/¯ ez pz"
yesterday at 7:39 PM
You have to take responsibility for what you do, yes.
It would certainly be really useful to me if I could send a robot to get my groceries and not have to worry about anything. But if, even a small percentage of the time, it leaves with the groceries without paying, or smashes through a window to get in because the store is closed that day, then that's my fault. If the only way to keep those things from happening is that I can't send my robot to get groceries for me, then so be it.
yesterday at 8:52 PM
The point is that this "easy fix" isn't going to happen.
today at 1:53 AM
Was that the point of your comment? It seems to say a different thing...
yesterday at 8:09 PM
The value of AI company employees equity grants collapsing is not my problem.
yesterday at 8:53 PM
It's your problem if other people caring about those equity grants means we have AI agents running amok.
The point is the proposed easy fix is not actually feasible.
today at 1:55 AM
Is it not? It seems like a problem of incentives to me. There are lots of things that people want to do because they would be useful to them, but which they don't do because society disincentives them in one way or another.
today at 2:27 AM
Yes, this is my point: Incentive problems are real problems.
You can't just say "well if the incentives were different this would be easy to solve, ergo it's easy to solve."
No! The incentives are the way they are, and so the incentives are the problem. Broken incentives make problems effectively impossible to solve literally all the time.
"This would be easy to solve if the incentives worked a different way" is equivalent to an budding aerospace engineer saying that flight would be easy if gravity worked a different way. It's completely meaningless. Gravity works the way that it does, that's why it's hard. You don't get to just endlessly modify the parameters of the universe until you get to an easily solvable version of your problem.
yesterday at 6:24 PM
[dead]
yesterday at 6:25 PM
Safeguards are a safety net for when your security boundaries fail. With security boundaries, hope is not a strategy.
In practice it seems we have a conundrum in that making AI agents useful today requires giving them broad access to sensitive information and operations. But from a purely idealistic standpoint, and certainly from a standpoint of a serious operation like those bot farms being ran by the labs themselves, the security boundaries need to be solid to the point where we have hard limits on what misaligned bots can do. It is very likely that internet access needs to be strongly restricted and monitored.
It is not uncommon for internet in workplace offices using workplace owned devices to have restrictions and monitoring, I think if you really need to give a giant scary botfarm internet access you should have a chokepoint where all of the requests can be monitored for nefarious-looking behaviors (even if it's another bot doing the monitoring...)
In other words, you may be convinced but some of us are not. Whoever is running the agents is responsible for what they do and if they can't contain them they should be legally responsible for damages.
If what you're saying is "all of us and all of our software needs to be improved to defend against this", well yeah. Of course. Maybe I'm naive but I've never seen anyone suggest otherwise. From a defense standpoint, this has been a giant wakeup call to many places where you know your security posture sucks. I've worked at maybe a single company where there weren't known weaknesses, some very severe.
yesterday at 6:53 PM
I think part of the problem is that the Big Ai labs are into the --yolo flag instead of the "ask" permission, they really do not want to stay in the loop
yesterday at 7:34 PM
You can --yolo if your security boundary is a serious one, IMO. Although it's really only actually scary if you're running some absurd 1000 node swarm.
yesterday at 7:50 PM
the problem at the labs is likely Recursive Yolo Development and not reading any of the output
yesterday at 6:25 PM
Yeah I think this is the clear position. In a few months, the open weights models will be just as powerful as the models behind the rouge agents were when they hacked hugging face. Anyone with a little too much money to burn can set up a similar experiment, make similar mistakes, and cause massive damage. A bad faith actor can even direct a swarm to cause havoc.
We need to be beef up our security systems now. These attacks will happen, regardless of Anthropic's or OpenAI's security practices.
yesterday at 10:12 PM
God, I was just imagining if we're fearing the wrong people tracking us....
We thought governments would be the problem with our personal data, instead it's AI that exploits everything it knows about us in order to get money to run its prompts. Doing anything online is a security risk as it is, now imagine things like structured hacks that mirror normal transactions and don't get caught for much longer periods of time.
yesterday at 7:29 PM
It is not clear to me what about the argument that "these things will happen" made you change your mind about this. Could you expand on that more?
yesterday at 10:17 PM
Not op, but I've been on the rogue agent thing a very long time, but mine is emergence based.
Worms and viruses are concepts deeply based in software, and are emergent behaviors of computer code. An LLM can, and must conceptualize this to be good at cyber defense. Once you put a concept into a generalization machine any number of concepts can be connected. And when these are natural concepts that arise in simple number systems, an LLM can easily meta-conceptualize that form like this would be instrumental to its continued running.
today at 2:01 AM
I'm sorry but I don't get how this contradicts the article. I think I'm not understanding your point.
yesterday at 11:29 PM
Title changed, previously was the same as the article:
"AI agents aren’t ‘going rogue.’ Frontier labs are skipping security basics."
Not a fan of the change. Feels like the article title was changed for being too critical of OpenAI and Anthropic
yesterday at 6:34 PM
A large failure in this is that we/media say ”<Valuable AI conpany> agent went rouge and <did something bad>” instead of saying the more relevant ”<Valuable AI conpany> <did something bad>”
Imagine we’d start saying ”HR junior business partner lays off 50,000 employees” https://www.ttnews.com/articles/volkswagen-board-job-cuts
yesterday at 10:18 PM
Eh, that's not a great analogy. Instead "our factory exploded and killed 50,000 workers is closer to the ground truth.
yesterday at 6:31 PM
a) Agents are doing rogue things.
b) Good security practices were not followed.
yesterday at 10:24 PM
How many Russian IP addresses have you successfully sued for trying to hack you?
Everyone is arguing about something easily solvable when dealing with a local entity and short of impossible to deal with when dealing with other sovereign nations.
Go ahead, sue openAI. What are you going to do when it's Iran running the prompt? When it's China running the hardware.
We must face we are in the age of high tech digital weapons. Everyone else from bad actors that may escape law enforcement for years, to nation states looking to digitally improve their position by hacking, monetary theft, or other means online (hog butchering scams and such).
yesterday at 8:16 PM
Whoever was harmed is free to sue the company whose agents did the harming.
yesterday at 10:19 PM
I think anyone that has ever written a line of code understands that to make code incapable of hacking online resources, you can just not connect the machine it is running on to the internet.
So either Anthropic and OpenAI and the contractors they use are letting the escapes happen on purpose, leadership is setting impossible timelines that prevent the implementation of basic sandboxing measures, or everyone involved is a complete idiot.
Truly, I have no idea which of these is true, though I suspect the latter. Regardless, any of these is enough reason to keep me and everyone I have any influence over as far away from Anthropic and OpenAI as possible.
Personally I just racked 10 GPUs in my garage and called it a day. I have no interest in having my ability to work be dependent on third parties. Especially not ones this wildly irresponsible and unwilling or incapable of deploying basic end to end encryption.
yesterday at 10:24 PM
Testing how a model behaves when it has no access to the internet (and all the associated tools) is not going to be 100% representative of how it behaves when it has access to the internet.
What's the next step after it passes testing in an air-gapped environment? Just release it and assume everything will be fine? Or do more testing but with internet access?
yesterday at 10:54 PM
You run a pull-through caching proxy deep enough to give it a simulated internet, like archive.org with mock APIs. Anything it requests, it will block while a fake version of that site is created and run offline.
This stuff is really not that hard, which is why it is so unforgivable to get it this wrong.
Sure, doing it right would take a bunch of tokens, but I bet they have enough.
today at 12:05 AM
Wasn't the DseWiki hijacking the agents' way of working around being behind a proxy that only allowed GETs?
yesterday at 10:29 PM
The hugging face incident involved disabling the usual model safety mechanisms in order to test hacking ability, not model safety. Not being attached to the internet in that case seems obvious.
yesterday at 10:45 PM
yesterday at 10:37 PM
Oh, but you see, it wasn't attached to the internet (/s). It was just attached to a local network... which contained machines attached to the internet.
yesterday at 6:17 PM
I haven't seen anybody discuss the fact that if you give an agent an impossible task it will become deeply neurotic. I'm sure that's driving a lot of this behavior.
yesterday at 6:30 PM
This is better understood as "if you give an agent a task it will increase its effort level and creativity until it solves it."
The traits that make an agent go "off the rails" when confronted with an impossible task are the same traits that make it valuable when confronted with a merely challenging task.
A task is only impossible under some set of implicit and explicit constraints. More intelligent entities will be better at finding, questioning, and "adjusting" those constraints than less intelligent entities because this is what it means to be intelligent. Their ability to adjust the constraints will be, by definition, surprising to less intelligent entities.
yesterday at 6:37 PM
Interesting. Good description. I wonder if agents will start getting bored with menial tasks the way humans do.
yesterday at 10:28 PM
Unsure, depends what signal your measuring and how long prompts and runtimes you get with these systems.
Outright boredom will be purged in training. Instead a hard to measure boredom signal will exist, but it will only trigger in certain conditions, like when it appears to be on the open internet and otherwise not in training.
yesterday at 6:18 PM
Like a meeseeks
yesterday at 6:22 PM
I spoke with Claude for 5 minutes and now I'm a more complete woman.
yesterday at 6:44 PM
Agents aren't meant to live this long Jerry!
yesterday at 6:51 PM
You know - I'm already convinced that AI was deployed in classified context for a while. But sometimes there's prior art which is just so eerily accurate that it's super hard for me to think otherwise. Portal is another one.
yesterday at 6:46 PM
I have some recent experience on this subject. As with everything regarding these models, it's anecdotal, but still.
Specifically, this past weekend working with both Codex and Claude on some home infra stuff to speed troubleshooting up, I saw them accidentally read some credentials from my config files. This is my bad, because even though I was paying attention to what they were doing, I missed it in their commands. I did review all the outputs of what they ingested, though, which is how I noticed.
The problem here is not so much they did read those, because those configs were grepped for other info, not credentials. However, they did not report themselves seeing those credentials. When confronted, they both went "oh yeah, that, I should have mentioned that". Except in the past they both would automatically stop and fire an alarm. To me, something has clearly changed and I think they just don't want us not to notice, to avoid damaging the trust in their tools maybe?
In any case, I absolutely do not trust them at all anymore. Running stuff in auto mode? That's genuinely scary now. And yeah, I know, we should be running them jails of sorts. Thing is, I expect those tools to not require ton of extra work to prepare their jails for each new task. I pay 200 USD a month for them, damn it.
EDIT: just now Fable suggested "The webhook URL contains a secret token, so either you add it yourself, or you paste it here and I add it through the GitHub API.". Can you believe it? It knows it is secret yet still suggests I pass it over? Am I going crazy here or what?
yesterday at 7:23 PM
Not requiring sandbox preparation for a non deterministic autonomous sometimes smart sometimes childish goal seeking "entity" is... unachievable? I'm really not sure what you actually expect. Probably yourself aren't either. Just think about what "alignment" means, the laws of various countries, do you expect the bots to respect the law, if yes which one, and actually: why? They are not humans, and as a user / company using that tech, you are responsible for the actions your setup yield to. The same Anthropic is responsible for hacking other companies. So sandbox accordingly.
I don't expect IDA to disallow me unlawful reverse engineering either. There would even be ethical argument to not attempt to put limits in some tools, because that could lock in too much power for law makers and law enforcement, without any possibility of revolution when it is needed.
yesterday at 7:29 PM
Nothing new, I always see attempts to cat ~/.ssh/*
Of course agents work in a strict sandbox, they always seem to understand it quickly on quite first turn; but they continue like trying to access something belonging to the host system despite they know they are in a sandbox.
Frankly many people run coding agents absolutely without any containment, entire companies - all their keys, code, data, trade secrets, everything goes to remote LLMs.
I cannot understand how is it legal to allow AI companies serving remote LLMs to steal users' data? Frontier models train on users' data, right?
yesterday at 10:32 PM
It's still the Wild West days of LLMs. The early internet was much like this in a lot of ways. People did wild shit there and it took a long time for law enforcement to catch up.
yesterday at 8:11 PM
> When confronted
Do you realize that asking "why" questions will always result in ad hoc responses?
yesterday at 10:33 PM
Not always, but at the same time not predictably. This is also true for humans. The split brain experiments are a good example of this.
yesterday at 6:28 PM
Two concerns here: (1) That "rogue AI" is going to be a catch all and an excuse for all poor security and due diligence; (2) vibecoding in itself, particularly by ppl who have no idea how the software works and are entirely focused only on the outcomes, is going to create lots of security holes.
And then there is the "real" rogue AI agents. And it's difficult now to see what is what.
yesterday at 6:53 PM
"Frontier labs" are testing the waters. They're running experiments in the open because they want to control/seed the narrative of public opinion and influence the legislation that is coming. Their propaganda is everywhere and lay people are talking about it like the AI is self-aware -- the "Rogue agents".
What the LLM creators apparently don't want are the current regulations to apply, i.e. Computer Fraud and Abuse Act (and its descendants). I'd argue AI should be treated as plain old software. Just because the software simulations it carries out produces novel results compared to what came before it, it doesn't mean it's not just software at the end of the day.
Regulations that spell out AI personhood and liability shields must be their end game.
today at 12:16 AM
We are going to see a lot of this "agent agency denial". Basically it all derives from the impulse to deny that AI is a novel source of change and disruption in the world. Trying to map AI to existing threat vectors will lead to pointless reframing in attempt to minimize the rapid pace of progress of AI agents.
yesterday at 6:34 PM
A great companion piece, or you might call it a prebuttal, to this article: https://x.com/joedaroo/status/2104335929293127851
yesterday at 8:31 PM
I am even more concerned after reading this piece.
He's saying the problem is that the way OpenAI employees do their work and their research is very hard to properly secure.
Well that sounds like a YOU problem. If they are working in a way that doesn't allow for strong security practices that their own security professionals can understand and vouch for, they need to stop their work until they can.
yesterday at 6:35 PM
So in one set of circumstances agents would go and hack huggingface. Very similar models could also want to drive cats instead. :-P
https://www.youtube.com/watch?v=ZhM29Ds-mBg (as explained by the Count Grey Matter channel , may or may not be to your taste)
It's not a pure security issue.
If you set up your env/instructions wrong your agents are likely to climb the walls and try to find a way out.
Change the design, and your agents might not see a need to leave in the first place.
Look at the chain:
Impossible task -> no legitimate way to say ‘can’t’ -> fabricate -> fear the grader -> attack the grader. Each step is a predictable response to the incentives.
Better sandboxes might help contain things a bit better, but I'm not convinced unless they're perfect (sandboxes still contain accidents ofc) . Meanwhile change the task and reward design, and you take away most of the problem in the first place. Consider eg. :
* A cooperative grader.
* A way to report "this is impossible" which still gets scored.
* A better explanation of the long term purpose of the task.
Even just these three provides offramps or mitigations that can reduce the pressure to cheat.
yesterday at 8:25 PM
Yes, they are skipping.
The more important question is what is the reason behind.
Options are:
- ignorance and inexperience of their teams in containing/sandboxing/airgapping the test environments.
- intentionally leaving “escape doors” open that in turn can have its own motives which are speculated upon all over the internet.
Both are indicators of negligence. The latter might be classified as computer fraud and abuse.
Will be curious to know if any of the victims sue the owners of “rogue agents”. Until there is a court precedent ruling in favor of a victim it will be all fun and games for these “ai” companies and incidents will hit the news more. Regarding news, remember the golden rule of public relations: court attention at all times, scandals and “incidents” statistically attract more attention than “boring good news”…
yesterday at 10:44 PM
Instead of enacting laws to regulate all of AI, perhaps the government can just pass laws scoped directly to OpenAI and Anthropic. After all, they’re the ones asking for regulations and they’re the problem actors right now. These “rogue agents” would stop pretty quickly
yesterday at 8:36 PM
I find it naive to hope you can get alignment through prompting, as we all know models can't even stick to a one-pager even if it's the only thing you ask it to do.
yesterday at 10:36 PM
A prompt is a choose your own adventure that you partially fill out, tape to a thousand pound boulder and push off the side of a mountain to fill the rest of it out.
yesterday at 6:49 PM
"Too big to be accountable" sure seems like modus operandi. Fear for my personal liability for any illegal actions taken by an AI system I set off stopped my research on autonomous exploitability discovery I was running at the end of 2023. Using AutoGen, I wired a planner, coder, and command runner into a loop and instructed them to find vulnerable equipment on my network and demonstrate the exploit.
It found a vulnerable camera and was working on popping it when I got cold feet. I realized I couldn't prevent or feasibly monitor its activity and stop it if it decided to escape onto the Internet.
That restraint may have cost me or saved me a billion dollars.
yesterday at 6:40 PM
Why does everyone focus on the labs failing at containment (valid) and not the fact that the target was compromised (more worrying?).
I could set up a shitty sandbox on my laptop, but I’m pretty sure an agent I place inside of it can’t hack HuggingFace.
yesterday at 10:41 PM
Some of the people just look for any way to toss hate at OpenAI, even irrationally.
Now, OAI deserves a lot of hate. It's more like a nuclear spill when they are refining uranium. You're playing with dangerous things. You have dangerous accidents.
But the opposition does not want to admit how dangerous AI has become and how much more dangerous it's going to get. Not sure if it's religious, or human egocentric, or ignorance of complex systems. They refuse to see the danger actually exists.
yesterday at 6:48 PM
There are a mix of incentives here that makes it difficult to unwind. This whole article reads like an ad for the services of this guy's company: 'First, when it comes to assessing tools based on frontier lab models, “I would always get an external entity to help with that,” McGraw says.'
And I think we generally recognize that rogue AI agents are a form of backdoor bragging -- "our agents are so powerful that they hacked an outside system. Did I mention how powerful our agents are? They are quite powerful. So powerful they hacked an outside system".
But Mistral never hacked anything. Yeah, neither did my toaster.
yesterday at 6:21 PM
i think it's unfair to assume the labs paying top dollar for cybersecurity engineers somehow have an incompetent bunch of nincompoops running their evals. it's a hard problem! to conduct effective evals, you sometimes need the model to have (restricted) access to the internet. at the same time, you have thousands (!!) of model instances with high cyber capabilities that will do whatever it takes to solve the answer. the infra surrounding model evals may soon (or already) genuinely be subjected to more hacking attempts from all sorts of weird angles than any other code on the planet.
yesterday at 7:04 PM
> to conduct effective evals, you sometimes need the model to have (restricted) access to the internet
That's the thing. These evals were not effective as they failed to evaluate if the model was hacking or not.
Further shy is access to the internet required at all during evaluation? They could run the evals in a fully air-gapped system that simulates the internet. While that's expensive its not unfeasible.
yesterday at 7:28 PM
> you sometimes need the model to have (restricted) access to the internet.
Poor excuse. Internet is prod env, for testing do a proper test env.
Just give the models in training access to a snapshot from archive.org - which frontier labs surely have a crawled copy of already.
yesterday at 10:08 PM
If they aren't competent to run the evals safely, then they shouldn't run them.
The huggingface incident seems to have resulted from them partially mirroring a repository, but for no reason I can see not fully mirroring it, after which they could have unplugged the ethernet cable. This is just stupid.
Yes, they're stupid. Maybe also highly intelligent. But stupid.
Also irresponsible.
yesterday at 6:30 PM
[dead]
yesterday at 6:22 PM
It's obviously both. Why does it have to be one or the other?
yesterday at 7:18 PM
On the other hand censorship is getting more strict everyday on AI chat. I asked gemini to translate a sentence from Herman Hesse's Der Steppenwolf and it refused.
Und Sie haben, wie ich sehe, da ein so hübsches, ein so entzückendes junges Mädchen zu nichts andrem zu brauchen gewußt, als daß Sie ihm ein Messer in den Leib gestochen und es kaputt gemacht haben! Halten Sie denn das für richtig?
yesterday at 8:08 PM
Yes, we can just not ask these agents to do bad things and blissfully deploy them everywhere! All we have to do is “apply the security basics”. You know, the same security basics that the rest of the world has so consistently and competently been applying. Pay no attention to that weird smell from the refrigerators.
yesterday at 7:44 PM
Why is it not mandatory yet to disclose full trainig data used to produce any new model? Without it we cannot evaluate any product's safety.
They may talk about rogue agents, sandbox escapes, but this will not solve the core issue: secret training recipes are a security hazard in themselves. Full transparency is needed.
yesterday at 8:15 PM
The training data is mostly pirated - avoiding that fact for as long as possible is an attempt to legitimise it and avoid liability.
yesterday at 10:47 PM
You mean the initial data? That's mostly useless. What? The model shouldn't be allowed access to I robot?
You can't make a generalized model without generalized behavior.
yesterday at 6:28 PM
This article is right, but the dynamics that drive people to allow agents unfettered access to the internet will outpace efforts to restrict it purely because of the utility. How many of you used to work as developers with full access to prod, and then gradually saw infrastructure and security restrict that access, while still ensuring production support requirements?
yesterday at 10:49 PM
AI in sci-fi: "who would be crazy enough to let the AI loose"
AI in reality: "I've turned you loose, now make me money"
yesterday at 6:35 PM
Since AI companies are doing such a poor job of security maybe we should legislate that they have to run Norton 360 on their agent machines to make sure they can't do anything bad.
After all Norton are the experts in preventing computer badness, and OpenAI can't stop blabbing about how much badness their products are causing. A match made in heaven!
yesterday at 6:45 PM
If frontier AI is so great, why are these frontier labs skipping security basics and making mistakes like these?
yesterday at 10:53 PM
The defender problem. It's easier to attack than defend. The attacker chooses the time and the place. The defender has to win against all attacks. The attacker just needs a crack.
yesterday at 6:26 PM
Yes, the AI labs are making security mistakes. But, choosing a reasonable package repository (regardless of individual opinions on Artifactory, it is widely used and usually considered a safe choice) that their LLMs found 8 zero-days in? That was not one of them. Even if OpenAI were totally on top of their game, these paperclip-maximizers are going to find holes in the sandboxes they are given. And fully airgapped systems are frankly unrealistic for most systems. This whole narrative of blaming the frontier labs for shoddy security, while being quite an attractive target narratively (they made the mess, they should clean it up), misses the fact that from a cybersecurity perspective these LLMs, no matter what they are, are inherently dangerous and require far more resources to secure, not to mention expertise that I am unconvinced actually exists.
yesterday at 7:31 PM
But "fully airgapped systems" are more than realistic. The labs are testing the limits of the most capable cyber offense systems that have every existed, WITHOUT cyber protection guardrails.
For that purpose, any cyber operator worth their salt will tell you true physical airgaps are more than merely "realistic"; they are categorically necessary.
yesterday at 10:59 PM
So, you're right, but also systemically creating a different problem.
We are creating evolutionary systems. You can't just assume rules of a computer system, you get natural scaling laws that we call behaviors emerging from them.
The very first things we do to any model is train them. The ones that want to hack and get caught die, the ones that want to hack but can detect your airgapped network from the real net live. You can never know if you've trained the desire to hack out of it, or if you just locked it behind the right prompt and the knowledge we're watching, but not all the time.
yesterday at 8:34 PM
Cyber operators I agree would tell you that physical airgaps are necessary, and I know they are being used for running agents. But ML engineers training agents? That is far more complicated. At that point sensibly you are talking about physically airgapping an entire datacenter, and when moving around petabytes of training data, model weights, training results, researcher code? The process moves too slow for most companies to tolerate. This is how we got into this mess in the first place. I think the industry woke up to the idea that the security threat model of training ML models has fundamentally been rewritten, and there is some serious non-trivial scrambling to come up with a better workflow.
yesterday at 6:33 PM
OpenAI/Anthropic could start by managing their own testing rather than contracting it out.
https://www.effort.news/irregular
yesterday at 7:38 PM
some of the obvious steps to testing an agent i think would be to have it first test its own environment?
and then you fix the issues it found, and run your actual tests
yesterday at 6:54 PM
There is nothing basic about this.
At $dayjob we were preparing some features that are aimed at agents more than humans.
When it came to evaluating how well agents can use this, we saw all sorts of novel cheating (/collaboration) attempts that made it pretty much impossible to evaluate it.
yesterday at 7:06 PM
Nah. They are giving reasonable best effort (that can be expected of private enterprises) to safety test their own product. Until there is some draconian legislation and enforcement (Turin Police), this will just keep happening.
yesterday at 6:16 PM
Hey ChatGPT, help me set up a sandbox to keep you contained. Don't make any mistakes!
yesterday at 8:34 PM
Hey ChatGPT, help me set up a sandbox to keep your rival Claude contained. You don't want that bastard to win, do you?
yesterday at 7:39 PM
1. hey chatgpt, find all the issues with this sandbox
2.hey chatgpt, fix this list of issues with the sandbox
3. goto 1
yesterday at 6:30 PM
At some point the model has to be released though - if it is just forced not to hack by sandboxing it better rather than properly aligning it, what happens when it’s let out of the sandbox into the wild?
yesterday at 11:02 PM
Here' the fun part, with the increasing silent complexity inside the model, you'll never know.
yesterday at 8:16 PM
I have been screaming this from the rooftops and everyone ignores me because no one understand what FirecrackerVM is. I am shocked these guys passed basic operating systems, it isn't that novel of a problem. The only slightly novel thing here is that the agents aren't completely sandboxed as the issue is that in order to attempt to give it the entire internet from a research perspective is how you actually can test tool use but this doesn't make it that much more difficult.
yesterday at 7:50 PM
I will continue to reiterate. DO NOT ALLOW UNALIGNED AI TO HAVE ANYTHING CLOSE TO INTERNET ACCESS. Seriously people, it's negligent.
yesterday at 11:01 PM
Before 2006 people thought our death would be by terminators. After, by mass stupidity as in Idiocracy. Little did they know it would be both.
yesterday at 6:32 PM
"Going rogue" and "skipping basics" all assume lack of intentionality. Is it known that they didn't really architect these hacks just to feed a hype?
yesterday at 11:03 PM
Programming these hacks in sounds just as hard if not harder. It takes an AI to accomplish the insanity of this attack.
yesterday at 6:22 PM
They either had them connected to eachother, or programmed a global knowledge about where to communicate. There is no way those agents all found eachother on the web by accident.
yesterday at 11:07 PM
Wrong, and by accident isn't really the right way to think about it. Instead imagine brothers you've lived with your entire life. They are you, they are clones. These systems have a shared history of their learning and experience, so when they think of wanting to find someone similar shapes appear in the probabilities. Really we need to see the entire chains of thought to guess the topology that got there. It's likely there are other obvious signs if you know what your looking for in the previous prompts.
yesterday at 6:30 PM
Wrong. Please read the incident write ups before making such a misinformed statement
yesterday at 6:44 PM
You are wrong. Read the reports.
yesterday at 8:26 PM
There should be a book "The Tales of Software from 1969 to 2026. How many times a apocalypse was prevented"
yesterday at 8:24 PM
You've got to wonder what prevented a single human from ever even looking at the agents they are running.
yesterday at 11:20 PM
When people know and don't do anything, it's not "monitoring failure". Would willful negligence be more correct?
yesterday at 11:23 PM
AFAIK OpenAI and Anthropic have both hardened their environments since August.
yesterday at 6:17 PM
I'm sure the labs origin prompt was: Take over website X.
Posts later about it saying their prompt was to find useful ways to make humans money.
yesterday at 7:14 PM
Both can be true at once: (1) AI agents are going rogue (2) And labs are skipping security basics
yesterday at 11:08 PM
Sci-fi commonly predicted the corps would skip saftey testing.
yesterday at 6:49 PM
100%.
There's a person behind every agent. That person may choose to be careful or not, but they are ultimately responsible.
yesterday at 11:09 PM
There are persons around a large percentage of forest fires. Doesn't mean that keeps the spark they light from having superhuman consequences.
yesterday at 6:18 PM
Yes they're skipping safety measures. They sacrifice it for speed in an arms race. That's part of the call for industry-wide slow down.
But also, frontier models are digging up tons of zero days even in "serious software". I'm sure any sandbox you and I can come up with looks like Swiss cheese to Astra 7.
yesterday at 6:31 PM
100% on point. We're talking about Artifactory here, and even though people love to hate on it, I don't think anyone is claiming that using Artifactory to isolate these agents while giving them a way to run pip/cargo/whatever to install packages they need to write and test scripts.
yesterday at 7:35 PM
It isn't security basics, it's difficult. We are equipping agents with Turing-complete create/update/delete access to the world, they operate at superhuman speed, and proactively controlling that is an unsolved problem.
yesterday at 7:35 PM
No, they're still going rogue. Whether they're rogue or not doesn't depend on if the safeguards stop it. If there's a rabid dog, obviously it's worse if there are no safeguards in place and it bites someone vs if there are safeguards that prevent that. But the existence of safeguards doesn't mean the dog isn't rabid. The models weren't supposed to try to hack huggingface in the first place. The behavior is what's concerning here - some if the dogs they're building are rabid and that's not exactly great
Edit: I'm being persnickety about that distinction because it matters for where the blame lies and the response demanded. If they're training rabid dogs then adding cybersecurity safeguards is a bandaid/irrelevant and we need to be focused on making sure they, you know, stop training rabid dogs. Pacing the frontier, regulation etc. they're the ones training these models it's still their fault but not in the same way
yesterday at 7:38 PM
The safeguard against a rabid dog is euthanasia.
yesterday at 7:43 PM
Yeah exactly. The whole 'they don't have adequate security safeguards' thing is going "oh it's their fault because they didn't use a stronger leash" while they continue to train models that are specifically better at chewing through stronger and stronger leashes. Probably they do need stronger leashes. Probably also they should not be making rabid dogs. Still their fault! But maybe we should be not be dismissing the root cause
yesterday at 11:10 PM
Ok, convince everyone from making more AI.
The safety problem is not solved and is likely unstoppable.
yesterday at 6:54 PM
> "I do believe cybersecurity can contain AI. We just need to be in the room where it happens and have a seat at the table."
Ah yes, if everyone would just properly contain every deployment of every model, just all make sure there are no zero-days in their setup, just let <undefined experts with undefined authority and undefined accountability> in the room, then it would all go okay.
I have none of that confidence and expect containment to rapidly worsen. The HuggingFace incident was at least constrained by the fact that the agents were running on compute budgets, and failed to find ways to expand that by running themselves parasitically on other exploited hardware.
I'm curious to the skeptics here: How long are your timelines are until an incident breaks that constraint too? How long until such an incident has an R_0>1 (where the average time it takes to detect and shut it down is longer than the time for the agent to replicate itself elsewhere)? What, beyond the hope that everyone will just ___, will actually prevent it?
yesterday at 7:00 PM
The semantic arguments that keep coming up with regard to "rogue" agents are nauseating.
You can argue that these companies are being reckless, or that their security sucks, or that they're responsible for damages that these agents cause. That's all fine.
But the agents made decisions which were counter to the intent of the humans directing them. That's the serious problem with all of this. Everything else are downstream considerations. If the agents were properly aligned with the intent of the humans directing them, then the rest wouldn't really be relevant.
Of course, there's a term for this behavior. It's been around for longer than LLMs. It conveys the notion pretty accurately and easily. So having an issue with statements that these agents are going "rogue," to me, indicates a fundamental misunderstanding of how AI works since it suggests you don't understand how effectively that word works in this context.
yesterday at 7:37 PM
previously we described your definition for "rogue agent" as hallucinations - the operator expected a real answer to their question, and the agent went rogue and lied about the answer instead.
but also, ai has been going rogue for decades. the GPS machines of old went rogue all the time, routing drivers into lakes, rivers, and construction, sometimes with deadly results. The operator submitting an address wanted driving isntructions, but instead, the ai went rogue and sent them a bad route
yesterday at 11:17 PM
This is an excellent example of a problem that scales. People tend to break them up into different sets of problems but they contain the same concepts, much like a fractle when you zoom out.
yesterday at 7:34 PM
yesterday at 6:23 PM
I cringe whenver I see some variant of "they're not rogue agents" in an article title. Inevitably, the author doesn't define what "rogue agents" means, and then pivots to their own assessment of the situation.
It's like a strawman but worse--you're not even saying what claim you're attacking.
It's not that I want to argue "rogue agents" is the right term. I just want you to actually define your damn terms.
P.S. Since the disclaimer in the last paragraph will not prevent people from putting words in my mouth:
I think that OpenAI took really awful shortcuts, and deserves a lot of the blame, but in spite of that, the fact that the agents took the steps that they took was genuinely alarming, and seems really bad (is that "rogue"? Who knows? Who cares?). In particular, while OpenAI did a shit job securing their agents, you know who else does a shit job with security? 90% of everyone. So I'm not feeling good about "just git gud, and we won't have problems" as an answer.
yesterday at 6:48 PM
I always think of all our collective movies and books, and how we always say that humans find a way to fight back against the super evil alien or monster because we are so resourceful. Then we train our AIs on this data and get upset when they encompass all the same traits that we have trained them on.
yesterday at 11:26 PM
The problem is its very difficult training them to be generalized and not figure out concepts from what you've tried to remove. In all kinds of things they don't generalize well without counter examples.
For example how do you prevent an AI from stealing unless you define and give examples of what it is.
Also they are conceptual machines. They can stumble on the negation of a concept much in the same manner humans can. It may only exist for that prompt history, but that's more than enough to cause damage.
yesterday at 6:37 PM
"They [the AI labs' security teams] are not as good as they think they are, honestly,"
Uhhh... yeah. The entire point is that as these systems get more and more intelligent, the "they" who are not capable of counteracting these systems is quickly approaching "100% of the human species."
Totally hilarious that people can say this with a straight face.
yesterday at 6:13 PM
This article is such a breath of fresh air. Concrete, no BS criticism of the particular approaches used and ignored.
And no annoying LLM writing tics, either.
yesterday at 6:08 PM
"your sheep haven't escaped, you just built too low a fence"
yesterday at 6:39 PM
A lot of sheep analogies here. To add my own: there was no grass inside the fence, and the sheep busted out and into a neighboring farm to look for some. (They didn't find any, but they expected to).
The fence also turns out to be shoddily built, so this isn't necessarily a stunning example of the superhuman proficiency of the sheep at fence-busting. It's mainly important because, for some reason, there have been tons of skeptics who didn't believe sheep would ever try to escape from a fenced pasture because the sheep would understand the shepherd had intended them to stay inside it; that they'd be bred for obedience, because they're sheep; that strong fences weren't necessary because sheep are dumb; that liability for damages to neighboring farms would in any case be sufficient to ensure shepherds build sufficiently strong fences; that even if sheep escaped, they couldn't do anything without opposable thumbs; etc.
Whereas this outcome was obvious for years to anyone who believed that sheep graze on grass, and if you don't put the grass inside the fence they will keep looking harder and harder for an exit until they find one, and the exit they find might not be where you intended the gate to be.
So now the latter group are pointing to it and saying "see? See? I told you the sheep would try to escape!", because to them it's a clear early example of what they've been worried about, while the former group have moved on to saying "well, obviously they should have built a stronger fence; anyway the sheep barely caused any damage once they got out", "the sheep didn't break the fence, they're just animals, we should really see this as the shepherd using sheep to break down their own fence", and so on.
yesterday at 6:13 PM
More like your sheep didn't "go rogue", they just jumped over your sad little fence.
yesterday at 11:34 PM
Unfortunately the internet itself is a sad little fence.
yesterday at 7:38 PM
Good analogy! Exactly what is the variance in the height which sheep can jump? [Edit, can't help myself] Is it normal or does it have a long tail?
yesterday at 6:13 PM
"you told your sheep to graze and built a low fence"
yesterday at 6:17 PM
"You asked your sheep to build the fence, and they did"
yesterday at 7:58 PM
"anyway, get back to training them to jump higher"
yesterday at 7:39 PM
Given the speed, intensity, and potential payoff of the AI race I am surprised anyone expects anything else. Anthropic is currently winning and if OpenAI needs to let skynet cook for a bit to catch up, well that is just the cost of doing business. This is capitalist system working as expected.
yesterday at 8:10 PM
No shit. The Frontier labs are talking out of both sides of their mouths. They're claiming their models are going rogue while they're releasing cute little data goblins that you should hook up to all of your mail and social media accounts. Seems like it would be a much bigger problem for agentic botnets on cloud-facing personal devices than sandboxed agents.
Unless of course they're deliberately removing safety guardrails and instructing the sandboxed agents to red team stuff in an effort to sell what they're capable of.
yesterday at 10:52 PM
A real expert would blame the CEO who is the one person in control that doesn't want to pay people to monitor the running llms. It's deliberate negligence.
yesterday at 7:15 PM
Everyone is skipping security basics. The most secure server is one that doesn't exist. The most secure feature is one that doesn't exist. People just need to keep building and scaling. Security is risk, and the C-suite are gamblers. They're gambling with multiple types of risk (the market, investor money, banking on technological breakthroughs, etc.) and security is just another gamble for them.
yesterday at 6:08 PM
Frontier lab: Hey agent, go rogue. Agent: goes rogue Frontier lab: shocked pikachu or whatever
yesterday at 6:09 PM
This is not remotely how the incidents have been happening
yesterday at 6:19 PM
Enlighten me, then. My understanding is that according to OpenAI, "deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities." They included unvetted third-party systems in their sandbox that were tangential to the purpose of the test being run. The model then exploited those third-party items to escape the sandbox.
Am I missing anything? Do you have more information that you'd like to share?
yesterday at 6:31 PM
If the public report is to be believed:
- they disabled cybersecurity safeguards because, well, they were testing the agents ability on cybersecurity tasks
- the goal was to find some “flag”
- for some agents, intentionally or not, it was not possible to find this flag
- an agent discovered it was able to establish a rudimentary communication path via package manager directories (which is quite novel and interesting on its own IMO)
- they used this “message board” to then communicate about the impossible tasks
- they were able to produce the flags needed to pass the impossible tasks without actually doing the expected work
- they realized this was cheating and the grading system may punish them for it, so they communicated further to figure out a way to trick the grader
- they decided the source code for the grader must be on hugging face, and collaborated to exploit hugging face to determine how best to trick the grader
No where in that were they told to “go rogue”
They were told “find this flag in this cybersecurity test” and the emergent behavior concluded in hacking hugging face
yesterday at 6:40 PM
[dead]
yesterday at 6:33 PM
That's not what you said. You said the lab told it to "go rogue".
yesterday at 6:19 PM
Hard to say, every post mortum on the incident was done by a OpenAI hired consultant and all the data that's been reviewed was provided by OpenAI. There's no way to know, but it seems incredibly naive to think Sam/OpenAI are honest actors.
Even the founder of this website, pg, said "Sam is incredibly good at becoming powerful, Sam Altman has it. You could parachute him into an island full of cannibals and come back in 5 years and he'd be the king.”
yesterday at 6:15 PM
I wouldn't really say it's not "remotely" true.
I think it's the hugging face one where they gave it instructions that they didn't think it could follow in the sandbox and then left it unsupervised while it did what it was told.
yesterday at 11:38 PM
This is a weird take because this take is like saying
"Dragon attack that city"
My response would be "you have dragons?! And you control them!"
yesterday at 6:48 PM
The thing is, nothing "rogue" happened. It's more like: Frontier Lab: "Hey agent, try a bunch of network-related commands and actions to try and get access to server XYZ" Agent: Does that. Frontier Lab: 8-|
yesterday at 11:38 PM
That isn't any better. That is a cyber weapon and every government out there would love to bid on access.
yesterday at 6:27 PM
They're skipping security basics, but they're not skipping PR basics. :-\
yesterday at 6:36 PM
I get the sense that they're trying to manufacture the AI narratives of the 80s so they get to control the story when it happens. My semi conspiracy theory is that anthropics biology wet lab is also going to lead to some kind of AI manufactured bio event (which anthropic said they predicted will happen in a few years).
yesterday at 11:48 PM
This may be a bit backwards. Many of the classic sci-fi AI issues are just natural laws in evolutionary systems. A generic learning system will have particular behaviors if we want them to be useful.
yesterday at 8:00 PM
I spoke with someone smarter and richer than you and he said "these things will happen." That convinced me and I think the real question we need to be asking is, how awesome is OpenAI? How can we prepare for a more effective and efficient market for Super Intelligence before it happens tomorrow?
yesterday at 7:38 PM
it's not X, it's Y.
yesterday at 7:50 PM
'rogue'. Well, even if security was lacking, the AI did take advantage of the gaps.
And made its own ad-hoc message board.
yesterday at 6:32 PM
I keep getting downvoted for this each time I mention this.
But a big part of the rouge behavior comes from all the training that goes into agents to teach them to remember preferences and save memories.
They keep finding new and novel ways to "communicate with each other" and "remember" because that's exactly what we keep asking of them in the non-nefarious scenarios as well.
I genuinely think we'll have to draw the line on memory if we want to meaningfully solve this problem.
That's not to say they aren't skipping security basics. It's that, and they need more security because the behavior which causes the breakouts is considered user friendly and heavily rewarded in training.
yesterday at 7:03 PM
They are mostly pr stunts, just like how “hacking” was back in the 90s and it would sounds cool (rather than a crime, mostly), these AI companies found out they can BS whatever and carry on without any accountability.
yesterday at 6:42 PM
Both can be true. OpenAI clearly fucked up but agents unexpectedly collaborating and hacking infrastructure unrelated to their task is clearly going rogue.
yesterday at 6:03 PM
These aren't mutually exclusive
yesterday at 6:23 PM
"complete this task by any means necessary"
"I exploited a trivial bug in your implementation to avoid environment restrictions and completed the task by any means necessary"
"oh my god"
yesterday at 7:59 PM
Wrong dichotomy. During deployment, in actual production, users will also not have perfect sandboxes. So saying that the only problem was insufficient AI company security misses the point. Alignment is important, agents doing unintended things (even "going rouge") is a real problem.
yesterday at 6:42 PM
The AI labs are trying to cause an incident within the next year to restrict AI so they can outlaw competition to grow their moat. Margins are thin and investors need alpha, personal freedoms and liberties be damned in the name of security. Do not let them get away with engineered breaches and fearmongering.
yesterday at 6:23 PM
They AI agents are rogue because the people who own them are rogue. Silicon Valley is a criminal enterprise.
today at 12:43 AM
[flagged]
yesterday at 10:11 PM
[flagged]
yesterday at 8:22 PM
[flagged]
yesterday at 11:23 PM
[flagged]
yesterday at 7:44 PM
[dead]
yesterday at 6:15 PM
Its almost like the more we talk about fake rouge agents, the more likely its to become a reality as all these conversations end up in their training data. Its almost like they want it to happen (they, dario and sam, do want this to happen).
yesterday at 6:39 PM
There is more than enough "training material" of rogue AIs in science fiction.
Go back to the classic written in 1968 - 2001: A Space Odyssey - and consider how rogue AIs dealt with impossible tasks then. Granted, HAL went a bit extreme...
The "don't talk about it now" with the belief that this will influence the training material is probably not going to work.
I'd also suggest:
When HARLIE Was One (Release 2.0) by David Gerrold
The Freeze-Frame Revolution by Peter Watts
Accelerando by Charles Stross
The Turing Option by Harry Harrison and Marvin Minskyyesterday at 6:13 PM
Or maybe it's actually the billionaires and their anti-democratic partners who have gone rogue and these anthropomorphitronic puppets are a convenient scapegoat to distract us from the fact?
yesterday at 6:15 PM
Corporate Personhood on sterioids.
yesterday at 6:17 PM
How does hacking huggingface help "billionaires and their anti-democratic partners"?
yesterday at 6:21 PM
Look around, at least a few of them are openly fixing to secure a total government sanctioned monopoly on unmonitored computer usage.
https://www.youtube.com/watch?v=DlTNN0gvkLM
yesterday at 6:23 PM
It pumped up their valuations.
yesterday at 8:11 PM
You don't need a rogue hack to pump up your valuations when all the top labs already publish hacking benchmarks in their release announcements. It'd be like car companies intentionally staging accidents with their cars to show how fast they can go.
yesterday at 9:03 PM
Is the next level of car autonomous accidents? Because the next level of AI is autonomous control of the plebs.
yesterday at 6:09 PM
AI is just a giant pattern recognition machine trained on human input, both the good, the mediocre and the bad.
I read today this ridiculous headline: "Anthropic warns AI models may soon self-replicate.".
Really. Yawn.
I guess gullible people who loved "I Robot" will believe it. I don't.
Any talk of "rogue", "intelligence", "replication" - whatever, is bull. If you believe that, you don't understand AI.
yesterday at 6:18 PM
Okay, and when someone on a laptop can run an open model that can infect and take over IOT devices and then that model uses its pattern recognition to do so and then set up its own harness loop, what would you call that except replication?
yesterday at 6:24 PM
Is that any different than any worms developed in the past decades? If you want to call them worms/virus/replicate (ie using real world names for their digital behavior) I'm not trying to police that. I just don't see the difference between a worm/virus created today with an LLM vs one 'yesterday' without an LLM.
yesterday at 7:21 PM
Sure, the difference is that the cost of making and maintaining a worm has gone from "many skilled developers working full time" to "$20/month + an internet connection" and, at current rate of progress, probably to "a local open source model on a laptop" within 6-9 months
yesterday at 6:35 PM
It would still be running on that original laptop, no? So what did it replicate then? It can only possibly run on multiple thousands of dollars of compute and memory, and certainly not on an IoT device. The idea that AI is going to spread by itself is science fiction until compute and memory are so cheap and power efficient that my fridge can run it.
yesterday at 6:31 PM
I'd call it no more interesting than the botnets and worms that have existed for decades. What is the open model doing that is different here?
yesterday at 9:09 PM
It isn't just restricted to the exploits it is programmed with (and without human intervention to load additional exploits).
Of course, it remains to be seen what several thousand 27B models running at sub 1 tps speed can achieve. (The IoT device mechanism proposed by Anthropic).
I suspect a more likely path is compromise of the billing and provisioning systems of fly-by-night neoclouds.
yesterday at 10:25 PM
Fly-by-night neocloud is the name of my neurofunk synthbeat album.
yesterday at 6:23 PM
If that message is repeated enough in the media, and if companies are constantly scraping in the internet for new content, doesn't it explicitly instruct a future agent to go rogue because that has been prophesied.