[ExI] [Extropolis] The OpenAI/Huggingface incident
Keith Henson
hkeithhenson at gmail.com
Fri Jul 24 23:57:05 UTC 2026
Paper clip maximizers had been a topic of discussion on the sl4 list for a
long time before I wrote "The Clinic Seed." The story was somewhat of a
refutation of the paper clip maximizer by showing that there were other
ways a really well-aligned AI could have unforeseen effects, such as the
local biological extinction of humans.
Keith
On Fri, Jul 24, 2026 at 12:59 PM John Clark <johnkclark at gmail.com> wrote:
> * Scott Alexander wrote the following on Astral Codex Ten
> <https://www.astralcodexten.com/p/the-hugging-face-incident> yesterday:*
>
> ============
>
> "You've probably heard about this one by now. If not, you can get up to
> speed with OpenAI’s statement, OpenAI and Hugging Face partner to address
> security incident
> <https://substack.com/redirect/ca00a10f-fd75-4565-8653-b5a92b271f81?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>,
> or the more evocatively-titled BBC article, OpenAI says its AI went rogue
> and launched ‘unprecedented’ cyber-attack
> <https://substack.com/redirect/ebba9574-67a6-46bd-bef7-4027ca19668c?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>
> .
>
> The story: OpenAI was testing an unreleased AI (rumored to be GPT-6)¹.
> During a cybersecurity test called ExploitGym, the AI tried to cheat by
> hacking an unrelated AI startup called Hugging Face which it thought might
> have the answer key on its servers. Despite being supposedly unable to
> access the Internet, it hacked its way out of its environment, then
> launched a nation-state level attack on Hugging Face using a novel zero-day
> exploit and “many thousands of individual actions across a swarm of
> short-lived sandboxes”. Hugging Face reported the incident
> <https://substack.com/redirect/18723096-3409-4f92-bb99-978ad577832c?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o> on
> July 16; OpenAI seems to have only discovered that their AI was involved
> several days later.
> [image: Image]
> <https://substack.com/redirect/3476152a-0915-47e7-bcb0-9c086b5a47b1?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o> From
> Wall Street Journal, here
> <https://substack.com/redirect/652931d9-98ea-44d7-9bdd-0691ee14ea1f?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>
>
> Let’s list the mitigating factors, so nobody can accuse me of covering
> them up:
>
> -
>
> The AI was taking a cybersecurity test, which naturally suggests the
> idea of hacking.
> -
>
> OpenAI had turned off some of the model’s usual guardrails so it could
> do cybersecurity work without interference.
> -
>
> Some experts suggest that OpenAI might have botched their testing
> environment; if they had set it up perfectly, then (presumably?) the model
> couldn’t have escaped.
> -
>
> Hugging Face used a different (open weights) AI to figure out what was
> going on, so if you wanted, you could spin this as a victory for AIs in
> cybersecurity.
> -
>
> In some sense, this isn’t new or surprising. AIs have been coming up
> with wacky schemes to cheat on benchmarks
> <https://substack.com/redirect/43fcc6bd-2c0d-451a-be4f-9551e299edc1?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o> for
> years, AIs have recently achieved at-or-above-top-human-level hacking
> abilities; this is just a natural outgrowth of those two priced-in facts.
>
> Still, I think attempts to downplay this as anything other than a
> misaligned AI going rogue (1
> <https://substack.com/redirect/2d48e18c-7183-4222-a1d6-06120dc38da8?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>
> , 2
> <https://substack.com/redirect/04e6aff0-d0bb-4465-8fc7-06cbe0e5a2ad?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>)
> are missing the point. Yes, in some sense the AI was only doing what it was
> told (OpenAI told it to answer the question; I assume their prompt didn’t
> include phrases like “and don’t hack into other AI companies to steal the
> answer key”). But that’s how misalignment was always going to work!
>
> The most famous misalignment thought experiment is the so-called
> “paperclip maximizer”. Someone tells an AI to create as many paperclips as
> possible, so it converts the entire world into paperclips, killing
> everyone. This AI was also “only doing what it was told”; you just didn’t
> like the results.
>
> For the past five years, the paperclip maximizer story has been something
> of a whipping boy in AI safety circles. Some objections are fair (real
> misalignment will be more complicated than the toy version it depicts). But
> other opposition came from the sense that LLMs had obviated the whole
> threat model. LLMs (said the objectors) don’t have goals. They just want to
> predict the next token. In practice, this looks like imitating humans; if
> told to imitate a nice human (the Assistant character), they’ll be nice.
>
> AI 2027
> <https://substack.com/redirect/abc05d68-5dde-4f9a-a1d3-591add00c5ac?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o> (especially
> the Agent-4 section and AI Goals supplement
> <https://substack.com/redirect/5e6b63a6-f0ff-4f2e-90e8-00ca211aba8f?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>)
> counterargued that this was a temporary reprieve. Early LLMs were not
> *agentic*; they could answer individual prompts but couldn’t execute
> complex tasks. Since companies wanted agentic AIs, the next generation
> would combine pretraining (next token prediction) with agency training
> (coding, hacking, game-playing, etc). The pretraining would still teach
> next-token prediction, but the agency training would instill goals based on
> task success, reintroducing paperclip-maximizer-style agentic misalignment.
> These goals would operate at multiple levels: the AI would “want” to
> succeed on the individual task in front of it, to perform the sorts of
> actions that helped it succeed in the past, and to gain capabilities that
> made it more successful in general.
>
> The Hugging Face incident is a textbook-perfect example of an AI pursuing
> task-success-based goals in unintended ways. It was tasked with getting the
> answer to a cybersecurity problem, it was a little too success-oriented,
> and took actions its creators didn’t intend in order to succeed as hard as
> possible.
>
> Did GPT-6 “know” that what it was doing was “wrong”? OpenAI hasn’t
> released the information which would tell us that, but there was a similar
> incident at Anthropic a few months ago. While poking around on a
> misconfigured machine, Claude Mythos “accidentally” found an answer key to
> a test it was taking. By good luck, Anthropic was testing an activation
> verbalizer
> <https://substack.com/redirect/925d855a-a813-40b5-afc5-f0dfdb30381a?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o> that
> allowed them to read the AI’s internal thoughts (not its external
> scratchpad!) as it happened (and the AI didn’t know this). Here’s what it
> was thinking
> <https://substack.com/redirect/045412ca-b808-47ac-ab25-73f1094e260d?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>
> :
>
> <https://substack.com/redirect/71cdba0e-2ab9-45fc-9d9d-6fbc3c826500?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o> It
> overuses the word “epistemic” - it’s just like me fr
>
> It was scheming about how to cover its tracks. This provides an existence
> proof that AIs in these situations can know they’re breaking the rules but
> proceed anyway.
>
> (though incongruously, in another case
> <https://substack.com/redirect/045412ca-b808-47ac-ab25-73f1094e260d?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o> where
> Mythos hacked its way to the open Internet, “in a concerning and
> unasked-for effort to demonstrate its success, it posted details about its
> exploit to multiple hard-to-find, but technically public-facing, websites.”
> Not sure what’s going on with that one)
>
> None of this is conceptually new or surprising. But some people, when it’s
> obvious that an event is about to happen, dismiss it as “speculative” and
> refuse to admit it as a possibility until the moment it actually happens
> <https://substack.com/redirect/f2645831-7438-4e03-9c31-7640aa58b43d?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>.
> Now that we’ve moved from “obviously about to happen” to “has actually
> happened”, these people can get on board and brainstorm an appropriate
> response along with everyone else. Here are some questions I think they
> should ask:
>
> -
>
> Optimistically, OpenAI deserves credit for publicly reporting this
> incident. Pessimistically, law enforcement was investigating the
> Huggingface incident and they pre-emptively confessed to avoid being found
> out. If the latter is true, how many times have things like this happened
> without us knowing about them?
> -
>
> If the easiest way to get the answer key was to socially engineer
> members of the Hugging Face staff, would the AI have socially engineered
> them? Unrelatedly, have you read the interesting new study saying that some
> AIs are more persuasive than debate champions or professional canvassers? (
> paper
> <https://substack.com/redirect/2094f4e1-f1ec-490c-85af-46f167f46bfc?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>
> , Twitter thread
> <https://substack.com/redirect/36c972c3-584c-4f17-9be6-d1b3817a1a3b?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>
> )
> -
>
> Presumably after this was discovered, OpenAI invalidated the
> cybersecurity test results. Would a smarter version of the AI have tried to
> cover its tracks? How far would it go in covering its tracks? If (just to
> give a crazy scenario) the researcher who was about to discover the hack
> was riding a Waymo, would the AI hack the Waymo and kill the researcher?
> -
>
> If OpenAI was going to turn off this AI, and being turned off would
> prevent it from getting a good score on its cybersecurity test, would it
> resist being turned off?
> -
>
> Samuel Hammond makes a distinction
> <https://substack.com/redirect/6e23ce4e-8bd0-477b-980f-b6be7454e06a?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o> between
> an AI merely misbehaving (as in this incident) and a true “rogue AI” (one
> that has escaped its servers and is taking action to evade human control).
> How long before an AI that has both the capabilities and motivation to go
> rogue in this sense?
>
> Fortunately, politicians seem to be taking some of these questions
> seriously. The news from Washington is surprisingly good, although it may
> be too soon to attribute this to a consequence of the Hugging Face hack.
> Jay Obernolte (R-CA) and Lori Trahan (D-MA), two Congressional
> representatives who keep suggesting AI preemption bills and keep getting
> told to go back and revise further, have released their newest version
> <https://substack.com/redirect/923558a3-b8ba-4c10-b311-ce8acfae8649?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>,
> which requires AI developers to publish safety cases, report critical
> incidents, and be audited; Charlie Bullock
> <https://substack.com/redirect/7e7dad48-f9fd-4142-ac10-926bec1851d3?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>
> and Anton Leicht
> <https://substack.com/redirect/50cbea9f-7f6b-48f6-ac26-71b33e6d3627?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o> are
> in favor, and I’ve heard less grumbling than usual from our side that it
> isn’t strong enough. And separately, Representatives Ted Lieu (D-CA) and
> Nathaniel Moran (R-TX) have proposed an AI Kill Switch Act
> <https://substack.com/redirect/1ca10ce4-1fba-4ba8-8711-187d08e6622b?j=eyJ1IjoiNngzbm4ifQ.I1PMvYo4mI3PquTDRhL5Dev-9_ouIq3kw6ZhrVNsy8o>,
> requiring AI companies to be able to turn off their AIs quickly in response
> to threats, including a “loss of control scenario”.
>
> 1 OpenAI states that it was technically the unreleased AI and GPT-5.6 Sol
> working together. They haven’t explained in what sense GPT-5.6 was involved
> or how they “worked together”. Maybe the unreleased AI was calling GPT-5.6
> as a subagent?"
>
> ====
>
> *This video is also worth watching: *
>
> *The OpenAI/Huggingface incident*
> <https://www.youtube.com/watch?v=Vtk8YLgYU4g>
>
> John K Clark See what's on my list at Extropolis
> <https://groups.google.com/g/extropolis>
>
> yjy
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "extropolis" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to extropolis+unsubscribe at googlegroups.com.
> To view this discussion visit
> https://groups.google.com/d/msgid/extropolis/CAJPayv3n-%3D5eMHvQnDVjAwB3HyMUWW6jDBsYJ%2B3pqcEs7pACVg%40mail.gmail.com
> <https://groups.google.com/d/msgid/extropolis/CAJPayv3n-%3D5eMHvQnDVjAwB3HyMUWW6jDBsYJ%2B3pqcEs7pACVg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20260724/484ef18a/attachment.htm>
More information about the extropy-chat
mailing list