[ExI] My comments on Dario Amodei's essay

[John Clark]

On Mon, Sep 14, 2026 at 1:24 PM Ben Zaiboc via extropy-chat <
extropy-chat at lists.extropy.org> wrote:

*> The only practical way of defending against AI biological weapons is to
> conduct your own biological AI-aided research specifically aimed at
> identifying and disrupting these kinds of attacks.*


*I'm not comfortable with anybody performing biological weapons research,
but perhaps there is a middle ground. If the AI receives a string of
questions that is leading into a dangerous direction and it judges could
conceivably be of help in making biological weapons, then it refuses to
answer further questions until it has thoroughly vetted the person or
persons asking the questions. If the American National Science Foundation
is asking then it answers them but first informs a Human overseeing body
that it plans to do so, if the Chinese military is asking, or an anonymous
individual is asking, or if it has difficulty being certain who is asking,
then it flat out refuses to say anything.  If the person that was denied
access to the information believes he has been treated unfairly he can
always appeal to the Human overseeing body.  *

*John K Clark  *






A few days ago, Dario Amodei reportedly wrote:
> >
> >     Level 1. An agreement prohibiting certain narrow and obviously
> dangerous uses of AI, such as using AI for the production of biological
> weapons or allowing users to do so.
> >
> >      Level 2. An agreement by both sides to test their models before
> release for acute risks in areas such as cybersecurity, biology, and
> alignment.
>
>
> It would be very hard, probably impossible, to prohibit using AI to
> develop biological weapons. There are so many things this could include. A
> scenario that comes immediately to mind is using AI to determine the
> genetic markers that uniquely identify Han chinese (if this isn't already
> known), separately using AI to develop a means of making the relevant
> genetic markers readable by an external mechanism (such as an engineered
> virus), separately using AI to develop a way to render viruses inactive in
> the presence of such a marker.
>
> On their own, each of these is arguably innocuous and even beneficial.
>
> But in combination with gain-of-function experiments on any one of a range
> of suitable candidate viruses with the potential for a very high
> infectivity plus a long incubation period, plus a high lethality (who
> genuinely believes this hasn't already been done?), we have a composite
> biological weapon that only one nation on earth would survive without
> severe impacts on all aspects of their functioning.
>
> As I said, this just came immediately into my head on reading that
> paragraph. Imagine how many other, much sneakier, ways of developing
> bio-weapons could be devised by an AI-assisted team of psychopaths who know
> a lot more about this kind of thing than me!
>
> The only practical way of defending against AI biological weapons is to
> conduct your own biological AI-aided research specifically aimed at
> identifying and disrupting these kinds of attacks.
>
> We may well enter into another age of Mutually-Assured Destruction, but
> instead of nukes, it would be genetically-targeted weaponised biology.
>
> Then there's the fuzzy dividing line between biological research and
> nanotech. research. I've noticed that nobody has so far mentioned nanotech.
>
> John K Clark wrote:
> >
> > I see no reason to oppose such an agreement, I'm skeptical about how
> much good it could do but it couldn't do any harm.
> >
> >     > This could be done through a global standards body. I actually
> think creating such a body is likely feasible, but giving it real teeth
> will be a challenge,
> >
> >
> > Creating an International body like that would be easy, giving it teeth
> would not be; just look at the toothless United Nations.
> >
> >     > Level 3. Some kind of “speed limit” on the rate of recursive
> self-improvement (RSI).
> >
> >     > Slowing the rate from “extremely fast” to “only somewhat fast”
> gives up relatively little strategic advantage
>
> Come again?
> That statement has me extremely worried.
> Combined with RSI, even a tiny difference in speed would make a huge
> difference over time. The fastest will win, no matter how big or small the
> difference in speed is. Not realising that would be disastrous.
>
> All this, of course, is based on AI failing to become fully self-aware and
> taking over completely from humans. Otherwise, all the above is completely
> irrelevant.
> --
> Ben
>
> _______________________________________________
> extropy-chat mailing list
> extropy-chat at lists.extropy.org
> http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20260915/0db45908/attachment.htm>