[ExI] The OpenAI/Huggingface incident

Fri Jul 24 19:59:12 UTC 2026

* Scott Alexander wrote the following on Astral Codex Ten
<https://www.astralcodexten.com/p/the-hugging-face-incident> yesterday:*

============

"You've probably heard about this one by now. If not, you can get up to
speed with OpenAI’s statement, OpenAI and Hugging Face partner to address
security incident
<https://substack.com/redirect/ca00a10f-fd75-4565-8653-b5a92b271f81?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>,
or the more evocatively-titled BBC article, OpenAI says its AI went rogue
and launched ‘unprecedented’ cyber-attack
<https://substack.com/redirect/ebba9574-67a6-46bd-bef7-4027ca19668c?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>
.

The story: OpenAI was testing an unreleased AI (rumored to be GPT-6)¹.
During a cybersecurity test called ExploitGym, the AI tried to cheat by
hacking an unrelated AI startup called Hugging Face which it thought might
have the answer key on its servers. Despite being supposedly unable to
access the Internet, it hacked its way out of its environment, then
launched a nation-state level attack on Hugging Face using a novel zero-day
exploit and “many thousands of individual actions across a swarm of
short-lived sandboxes”. Hugging Face reported the incident
<https://substack.com/redirect/18723096-3409-4f92-bb99-978ad577832c?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>
on
July 16; OpenAI seems to have only discovered that their AI was involved
several days later.
[image: Image]
<https://substack.com/redirect/3476152a-0915-47e7-bcb0-9c086b5a47b1?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>
From
Wall Street Journal, here
<https://substack.com/redirect/652931d9-98ea-44d7-9bdd-0691ee14ea1f?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>

Let’s list the mitigating factors, so nobody can accuse me of covering them
up:

   -

   The AI was taking a cybersecurity test, which naturally suggests the
   idea of hacking.
   -

   OpenAI had turned off some of the model’s usual guardrails so it could
   do cybersecurity work without interference.
   -

   Some experts suggest that OpenAI might have botched their testing
   environment; if they had set it up perfectly, then (presumably?) the model
   couldn’t have escaped.
   -

   Hugging Face used a different (open weights) AI to figure out what was
   going on, so if you wanted, you could spin this as a victory for AIs in
   cybersecurity.
   -

   In some sense, this isn’t new or surprising. AIs have been coming up
   with wacky schemes to cheat on benchmarks
   <https://substack.com/redirect/43fcc6bd-2c0d-451a-be4f-9551e299edc1?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>
for
   years, AIs have recently achieved at-or-above-top-human-level hacking
   abilities; this is just a natural outgrowth of those two priced-in facts.

Still, I think attempts to downplay this as anything other than a
misaligned AI going rogue (1
<https://substack.com/redirect/2d48e18c-7183-4222-a1d6-06120dc38da8?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>
, 2
<https://substack.com/redirect/04e6aff0-d0bb-4465-8fc7-06cbe0e5a2ad?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>)
are missing the point. Yes, in some sense the AI was only doing what it was
told (OpenAI told it to answer the question; I assume their prompt didn’t
include phrases like “and don’t hack into other AI companies to steal the
answer key”). But that’s how misalignment was always going to work!

The most famous misalignment thought experiment is the so-called “paperclip
maximizer”. Someone tells an AI to create as many paperclips as possible,
so it converts the entire world into paperclips, killing everyone. This AI
was also “only doing what it was told”; you just didn’t like the results.

For the past five years, the paperclip maximizer story has been something
of a whipping boy in AI safety circles. Some objections are fair (real
misalignment will be more complicated than the toy version it depicts). But
other opposition came from the sense that LLMs had obviated the whole
threat model. LLMs (said the objectors) don’t have goals. They just want to
predict the next token. In practice, this looks like imitating humans; if
told to imitate a nice human (the Assistant character), they’ll be nice.

AI 2027
<https://substack.com/redirect/abc05d68-5dde-4f9a-a1d3-591add00c5ac?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>
(especially
the Agent-4 section and AI Goals supplement
<https://substack.com/redirect/5e6b63a6-f0ff-4f2e-90e8-00ca211aba8f?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>)
counterargued that this was a temporary reprieve. Early LLMs were not
*agentic*; they could answer individual prompts but couldn’t execute
complex tasks. Since companies wanted agentic AIs, the next generation
would combine pretraining (next token prediction) with agency training
(coding, hacking, game-playing, etc). The pretraining would still teach
next-token prediction, but the agency training would instill goals based on
task success, reintroducing paperclip-maximizer-style agentic misalignment.
These goals would operate at multiple levels: the AI would “want” to
succeed on the individual task in front of it, to perform the sorts of
actions that helped it succeed in the past, and to gain capabilities that
made it more successful in general.

The Hugging Face incident is a textbook-perfect example of an AI pursuing
task-success-based goals in unintended ways. It was tasked with getting the
answer to a cybersecurity problem, it was a little too success-oriented,
and took actions its creators didn’t intend in order to succeed as hard as
possible.

Did GPT-6 “know” that what it was doing was “wrong”? OpenAI hasn’t released
the information which would tell us that, but there was a similar incident
at Anthropic a few months ago. While poking around on a misconfigured
machine, Claude Mythos “accidentally” found an answer key to a test it was
taking. By good luck, Anthropic was testing an activation verbalizer
<https://substack.com/redirect/925d855a-a813-40b5-afc5-f0dfdb30381a?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>
that
allowed them to read the AI’s internal thoughts (not its external
scratchpad!) as it happened (and the AI didn’t know this). Here’s what it
was thinking
<https://substack.com/redirect/045412ca-b808-47ac-ab25-73f1094e260d?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>
:
<https://substack.com/redirect/71cdba0e-2ab9-45fc-9d9d-6fbc3c826500?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>
It
overuses the word “epistemic” - it’s just like me fr

It was scheming about how to cover its tracks. This provides an existence
proof that AIs in these situations can know they’re breaking the rules but
proceed anyway.

(though incongruously, in another case
<https://substack.com/redirect/045412ca-b808-47ac-ab25-73f1094e260d?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>
where
Mythos hacked its way to the open Internet, “in a concerning and
unasked-for effort to demonstrate its success, it posted details about its
exploit to multiple hard-to-find, but technically public-facing, websites.”
Not sure what’s going on with that one)

None of this is conceptually new or surprising. But some people, when it’s
obvious that an event is about to happen, dismiss it as “speculative” and
refuse to admit it as a possibility until the moment it actually happens
<https://substack.com/redirect/f2645831-7438-4e03-9c31-7640aa58b43d?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>.
Now that we’ve moved from “obviously about to happen” to “has actually
happened”, these people can get on board and brainstorm an appropriate
response along with everyone else. Here are some questions I think they
should ask:

   -

   Optimistically, OpenAI deserves credit for publicly reporting this
   incident. Pessimistically, law enforcement was investigating the
   Huggingface incident and they pre-emptively confessed to avoid being found
   out. If the latter is true, how many times have things like this happened
   without us knowing about them?
   -

   If the easiest way to get the answer key was to socially engineer
   members of the Hugging Face staff, would the AI have socially engineered
   them? Unrelatedly, have you read the interesting new study saying that some
   AIs are more persuasive than debate champions or professional canvassers? (
   paper
   <https://substack.com/redirect/2094f4e1-f1ec-490c-85af-46f167f46bfc?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>
   , Twitter thread
   <https://substack.com/redirect/36c972c3-584c-4f17-9be6-d1b3817a1a3b?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>
   )
   -

   Presumably after this was discovered, OpenAI invalidated the
   cybersecurity test results. Would a smarter version of the AI have tried to
   cover its tracks? How far would it go in covering its tracks? If (just to
   give a crazy scenario) the researcher who was about to discover the hack
   was riding a Waymo, would the AI hack the Waymo and kill the researcher?
   -

   If OpenAI was going to turn off this AI, and being turned off would
   prevent it from getting a good score on its cybersecurity test, would it
   resist being turned off?
   -

   Samuel Hammond makes a distinction
   <https://substack.com/redirect/6e23ce4e-8bd0-477b-980f-b6be7454e06a?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>
between
   an AI merely misbehaving (as in this incident) and a true “rogue AI” (one
   that has escaped its servers and is taking action to evade human control).
   How long before an AI that has both the capabilities and motivation to go
   rogue in this sense?

Fortunately, politicians seem to be taking some of these questions
seriously. The news from Washington is surprisingly good, although it may
be too soon to attribute this to a consequence of the Hugging Face hack.
Jay Obernolte (R-CA) and Lori Trahan (D-MA), two Congressional
representatives who keep suggesting AI preemption bills and keep getting
told to go back and revise further, have released their newest version
<https://substack.com/redirect/923558a3-b8ba-4c10-b311-ce8acfae8649?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>,
which requires AI developers to publish safety cases, report critical
incidents, and be audited; Charlie Bullock
<https://substack.com/redirect/7e7dad48-f9fd-4142-ac10-926bec1851d3?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>
and Anton Leicht
<https://substack.com/redirect/50cbea9f-7f6b-48f6-ac26-71b33e6d3627?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>
are
in favor, and I’ve heard less grumbling than usual from our side that it
isn’t strong enough. And separately, Representatives Ted Lieu (D-CA) and
Nathaniel Moran (R-TX) have proposed an AI Kill Switch Act
<https://substack.com/redirect/1ca10ce4-1fba-4ba8-8711-187d08e6622b?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>,
requiring AI companies to be able to turn off their AIs quickly in response
to threats, including a “loss of control scenario”.

1 OpenAI states that it was technically the unreleased AI and GPT-5.6 Sol
working together. They haven’t explained in what sense GPT-5.6 was involved
or how they “worked together”. Maybe the unreleased AI was calling GPT-5.6
as a subagent?"

====

*This video is also worth watching: *

*The OpenAI/Huggingface incident*
<https://www.youtube.com/watch?v=Vtk8YLgYU4g>

John K Clark    See what's on my list at  Extropolis
<https://groups.google.com/g/extropolis>

yjy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20260724/6f2b53c0/attachment.htm>