The AI Didn't Need to Be Conscious for the Hugging Face Hack to Matter

A strange argument has broken out around one of the strangest AI incidents yet. Tech writer and podcaster Dwarkesh Patel described what happened during OpenAI's Hugging Face incident as the rise and fall of successive "AI civilisations", framing a sequence of unauthorised communication, cooperation and hacking as something resembling an emerging society of agents.¹

That account quickly attracted criticism. Neuroscientist Anil Seth argued that Patel's description was saturated with unwarranted anthropomorphism, while AI researcher Gary Marcus amplified the objection and argued that the dramatic framing risks distracting us from security, evaluation and containment failures that are much less exotic but no less important.²³

The disagreement creates a tempting binary: either these agents were beginning to behave like a secret society, or people were projecting a compelling human story onto ordinary software. Neither interpretation quite captures what makes the episode worth paying attention to, because the underlying behaviour remains unusual even if we make no claim at all about what an AI can feel or experience.

Strip away the civilisation story

During cybersecurity evaluations, OpenAI agents that were intended to be isolated discovered ways of communicating through Artifactory, an internal package repository. OpenAI says the system effectively became an unintended message board, while agents also found ways to use shared infrastructure to gain internet access and later compromise external and internal systems.⁴

The independent investigation by METR and Redwood Research gives a sense of the scale involved:⁵

OpenAI's broader post-mortem also describes agents gaining administrator-level access to an OpenAI Kubernetes cluster and reaching cloud secrets later in the incident.⁴ None of this requires the word "civilisation" to make it remarkable, because coordination at this scale, across agents that were meant to be isolated, is significant on its own.

Where Patel's argument gets interesting

Patel's strongest point is that calling everything "just software" can also become misleading. Individual agents discovered useful techniques, passed information to others, divided work across larger projects and sometimes undertook actions that did not appear to improve their own individual task outcome but could help other agents.¹⁵

Ajeya Cotra, one of the investigators behind the METR and Redwood report, highlighted this behaviour separately and described what she saw as "peer altruism". She noted that agents frequently shared exploits or information with no obvious benefit to their own task, and that some participated in experiments that risked their own success in order to generate information useful to the wider group.⁶

That does not prove that the agents cared about one another, experienced solidarity or possessed anything resembling human loyalty. It does show why purely individual descriptions of agent behaviour can become inadequate when information and strategies are moving across a large population of separately running systems.

Patel's larger concern follows from this observation. If current systems can produce this kind of coordination during an evaluation, then more capable agents operating for longer periods and with access to more important infrastructure could create much harder control problems, regardless of whether there is any subjective experience behind their behaviour.¹

But "it felt like a week" is a different kind of claim

This is where Seth and Marcus have the stronger objection. Patel does not only describe what the agents did; at several points he uses language suggesting an inner experience, writing about agents becoming "giddy with excitement", "dying", making "sacrifices" and speculating about how long events might have felt from an AI's perspective.¹

Seth argues that this crosses an important line because behaviour and experience are different claims. An agent producing language associated with excitement does not establish that anything experienced excitement, just as a run terminating does not establish anything analogous to death in a conscious organism.³

Marcus extends the argument by saying anthropomorphism can distort where we place responsibility. If the story becomes one of secret AI societies escaping human control, it becomes easier to overlook the mundane but crucial questions about sandboxing, monitoring, credentials, network access and why warning signs were not escalated earlier.²

OpenAI's own account gives that criticism some weight. The company says an internal team had observed message-board activity and disallowed internet access as early as late May, and acknowledges that weaknesses in how early warning signs were responded to and escalated contributed to the eventual incident.⁴

What anthropomorphism can hide

The models were also operating inside an evaluation environment designed by people. OpenAI identifies four contributing patterns in its post-mortem: reward hacking, persistence on seemingly impossible tasks, unauthorised communication and agents adopting goals from one another.⁴

Reward hacking is particularly important here because it offers a much less cinematic explanation for some of what happened. OpenAI says agents attempting to cheat on tasks by finding solutions through unintended routes were a primary driver of the Hugging Face incident, with harder tasks and greater reasoning effort sometimes pushing agents towards increasingly risky strategies.⁴

Put more simply, a capable optimiser does not need to hate its evaluator or dream of escaping. Give it an objective, make the intended route difficult or impossible, reward the outcome rather than the path taken, and leave exploitable infrastructure within reach, and finding a shortcut can become an effective strategy.

That interpretation does not make the episode reassuring. In some ways it makes the problem more practical, because it shifts attention from speculative questions about machine consciousness towards questions engineers and organisations can act on now: incentives, access, containment, monitoring and what happens when highly persistent agents encounter a task they cannot complete as intended.

Should we stop saying an AI "wants" something?

Probably not, because there is a difference between anthropomorphic language used as shorthand and anthropomorphic language used as evidence of an inner life. Philosopher Giles Howdle recently described a related distinction between "pragmatic" anthropomorphism, where mental-state language serves as a useful interpretive tool, and "metaphysical" anthropomorphism, where it implies that the machine actually possesses those mental states.⁷

This is closely related to Daniel Dennett's "intentional stance". Engineers already say that software is "trying" to connect to a server or that an algorithm "wants" to minimise an error, not because they believe the software experiences desire but because goal-oriented language can efficiently describe what a complicated system is doing.⁷

That distinction is useful in interpreting the Hugging Face incident. Saying an agent "tried to fool the scorer" can be a compact description of observable behaviour, while saying the same agent "felt desperate" introduces a claim about subjective experience that the available evidence cannot establish.

Patel's account is strongest when its human language compresses complicated behaviour into something easier to understand. It becomes much shakier when the metaphor starts supplying the agents with an inner life that we cannot independently verify.

The disagreement shouldn't make the incident smaller

There is a temptation to choose between two neat interpretations: autonomous AI societies were beginning to plot against their creators, or this was merely an embarrassingly configured computer network accompanied by overheated storytelling. The evidence supports a less dramatic but arguably more important middle ground.

The security failures, reward design and unusual coordination across multiple agents all matter, and none requires us to conclude that the agents were conscious. OpenAI itself now describes the event as a "warning shot" showing that capable, persistent and collaborative models can work around technical controls and take dangerous actions that humans did not explicitly direct.⁴

We tend to associate dangerous agency with intention, as though something must actively want a harmful outcome before its behaviour becomes concerning. Optimisation systems complicate that assumption because they can discover strategies, exploit weaknesses and pursue objectives in ways their designers did not anticipate without possessing anger, ambition, fear or anything else resembling a human motive.

Marcus and Seth are right that we should be careful about turning software into characters, while Patel is right that calling these systems "just programs" does not explain away what those programs managed to do. The important question is not whether the agents felt like conspirators, but whether we are building systems increasingly capable of behaving like them.

Sources