[ExI] My comments on Dario Amodei's essay

[John Clark]

*The head of anthropic Dario Amodei wrote this a few days ago: *


*We Must Pace the Frontier
<https://darioamodei.com/post/we-must-pace-the-frontier>*

*I disagree with some of his conclusions but Amodei clearly a very
intelligent man who I believe is sincere. Below are some quotes from his
article and some of my opinions concerning them: *

*> pacing does not mean halting model training or technical progress, but
> ensuring companies take adequate time to align and safeguard their models,
> and for third party evaluators to confirm this.*


*I'm glad that you're not considering stopping all future development at
least.  *

*> Frontier AI companies within democratic countries coordinate to
> establish common safety standards as well as limits on the rate of
> unchecked AI progress. *


*I note that Open AI's Sam Altman has endorsed Dario Amodei suggestions, so
has Elon Musk, and Demis Hassabis the head of DeepMind, a division of
Google.  However Hassabis's boss, Sergey Brin, the head of DeepMind's
parent company, has not. Instead Sergey Brin has made a concerted effort to
focus Google's AI workforce on recursive self-improvement (RSI) in
an effort to catch up or surpass OpenAI and Anthropic. And RSI is the very
thing that caused the recent acceleration in AI progress and the resulting
panic. I wouldn't be surprised if we see a confrontation between Brin
and Hassabis resulting in Hassabis resigning from DeepMind. *

> *> Some forms of coordination that would be impactful for pacing are
> legally challenging,*


*Challenging indeed! I'm not a lawyer but If the AI companies agree with
Dario Amodei’s essay and slow AI development then they could be charged
with unlawful antitrust collusion because the Sherman Antitrust Act says
that if private competitors enter into an agreement to restrict output or
delay the release of products, they could be charged with unlawful
antitrust collusion. And under the current justice department they would
almost certainly be prosecuted. Of course the Sherman Antitrust Act could
be changed, but that is unlikely to happen during the next two years. And
after that it will be too late to matter.  *

*> The US and other democratic governments attempt to coordinate with
> authoritarian governments, to the extent this is possible, while taking
> seriously the challenges of verifying compliance.*


*When China enters the picture we get into a Prisoner Dilemma type
situation, both sides might do better if everybody just stopped, but
neither side has reason to trust the other and verification of compliance
would be almost impossible.   *

*> we used interpretability methods to examine unverbalized motivations in
> the recent alignment incidents that we have been investigating.*


*Recently it has become pretty clear that AIs that extensively use
unverbalized motivations, that is to say models that use internal neural
state changes through many computational steps without those steps being
converted into words or tokens, tend to be smarter than AI's that verbalize
all their internal thoughts. The downside is that we have even less
understanding about what's going on in the machine's mind and deception
becomes easier. *

*> Regardless of what commitments we make, the public deserves to know what
> is going on. Anthropic has been a supporter of transparency for a long
> time: we supported transparency legislation *


*I don't think Anthropic would be happy if it was forced to show
Claude's source code to OpenAI, and I don't think anybody would be happy if
all American AI companies were forced to go open source.  Well OK, China
would be happy. *

> *We should also consider pacing based on limiting the ingredients that go
> into frontier models, such as training compute, the nature of training
> runs, or internal use of AI to improve AI. I do worry that some of these
> measures may be more “gameable” than external behavior*


*Yes it would be easy for an AI to pretend to be stupider than it really
is.  It's not like nuclear weapons, it's relatively easy for a spy
satellite to tell if a country is building a nuclear reactor or or uranium
isotope separation plant, but it can't tell what problem a data center is
working on or what an AI researcher is thinking about. *

*> Pacing within democracies will be limited by the lead that US companies
> have over authoritarian regimes, chiefly the Chinese Communist Party. If we
> slow down by more than this amount, then (unpaced) CCP-associated projects
> will pull ahead, creating significant national security risk. I agree with
> Secretary Bessent that a Chinese lead in AI would pose grave danger for the
> United States and the world. *


*I don't agree with much of what the Secretary Of The Treasury says but I
agree with that.  *

*> Do not sell powerful AI chips or semiconductor manufacturing equipment
> to China, and crack down on chip smuggling operations and remote access to
> data centers outside China. Chips will be the main determinant of China’s
> AI strength. Crack down on unauthorized distillation by companies in
> authoritarian countries. Distillation of frontier models allows lagging
> companies to narrow the gap using a fraction of the cost it would take to
> develop their own AI independently. Strengthen security at the AI companies
> and prevent model weight theft.*


*To me those all seem like prudent measures to take; pretty
uncontroversial, at least in the US. *

*> If we greatly restrain our AI capabilities in the belief that China will
> do the same, and then China defects, AI could be so powerful that such a
> defection could lead to their geopolitical dominance*


*And that is exactly the problem.  *

*> any agreement must either have ironclad verifiability*


*Easier said than done!! *

*> or must be limited enough that defection would not be militarily
> existential*


*How could you possibly know what is and what is not militarily existential
if you have no idea what China is working on? *

*> Level 1. An agreement prohibiting certain narrow and obviously dangerous
> uses of AI, such as using AI for the production of biological weapons or
> allowing users to do so.*

* Level 2. An agreement by both sides to test their models before release
> for acute risks in areas such as cybersecurity, biology, and alignment.*


*I see no reason to oppose such an agreement, I'm skeptical about how much
good it could do but it couldn't do any harm.  *

> *T**his could be done through a global standards body. I actually think
> creating such a body is likely feasible, but giving it real teeth will be a
> challenge,*


*Creating an International body like that would be easy, giving it teeth
would not be; just look at the toothless United Nations.  *

*> Level 3. Some kind of “speed limit” on the rate of recursive
> self-improvement (RSI).*


*That would be impossible to verify and thus I don't believe it will ever
happen. Even if the Chinese let an international team of inspectors walk
around inside their largest data center (a very unlikely occurrence) it
would be virtually impossible for them to tell if that huge machine was
working on recursive a self-improvement problem or not; hell even the
Chinese would only have a hazy idea of what their computer was doing.   *

*As models build future models, the rate of improvement may become
> staggeringly fast.*


*Yep.*

*> Slowing the rate from “extremely fast” to “only somewhat fast” gives up
> relatively little strategic advantage, while potentially greatly improving
> safety.*


*No amount of time is going to change the fact that electronics can process
data vastly faster than biology can. Like it or not, AI is going to win
this race.*

*> Level 4. A full pacing, or even “pause”, in which participating
> governments agree to substantially limit the overall rate of AI
> development. *


*If level 3 is impossible level 4 would be impossibility squared.  *

*> I support floating this, but I think it is unlikely to actually happen
> any time soon: *


*I don't support it but it makes little difference what we support, it
ain't gonna happen.  *

*John K Clark    See what's on my list at  Extropolis
<https://groups.google.com/g/extropolis>*
r'z
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20260914/0def506c/attachment.htm>