[ExI] My comments on Dario Amodei's essay

[Ben Zaiboc]

A few days ago, Dario Amodei reportedly wrote:
>
>     Level 1. An agreement prohibiting certain narrow and obviously dangerous uses of AI, such as using AI for the production of biological weapons or allowing users to do so.
>
>      Level 2. An agreement by both sides to test their models before release for acute risks in areas such as cybersecurity, biology, and alignment.


It would be very hard, probably impossible, to prohibit using AI to develop biological weapons. There are so many things this could include. A scenario that comes immediately to mind is using AI to determine the genetic markers that uniquely identify Han chinese (if this isn't already known), separately using AI to develop a means of making the relevant genetic markers readable by an external mechanism (such as an engineered virus), separately using AI to develop a way to render viruses inactive in the presence of such a marker.

On their own, each of these is arguably innocuous and even beneficial.

But in combination with gain-of-function experiments on any one of a range of suitable candidate viruses with the potential for a very high infectivity plus a long incubation period, plus a high lethality (who genuinely believes this hasn't already been done?), we have a composite biological weapon that only one nation on earth would survive without severe impacts on all aspects of their functioning.

As I said, this just came immediately into my head on reading that paragraph. Imagine how many other, much sneakier, ways of developing bio-weapons could be devised by an AI-assisted team of psychopaths who know a lot more about this kind of thing than me!

The only practical way of defending against AI biological weapons is to conduct your own biological AI-aided research specifically aimed at identifying and disrupting these kinds of attacks.

We may well enter into another age of Mutually-Assured Destruction, but instead of nukes, it would be genetically-targeted weaponised biology.

Then there's the fuzzy dividing line between biological research and nanotech. research. I've noticed that nobody has so far mentioned nanotech.

John K Clark wrote:
>
> I see no reason to oppose such an agreement, I'm skeptical about how much good it could do but it couldn't do any harm.  
>
>     > This could be done through a global standards body. I actually think creating such a body is likely feasible, but giving it real teeth will be a challenge,
>
>
> Creating an International body like that would be easy, giving it teeth would not be; just look at the toothless United Nations.  
>
>     > Level 3. Some kind of “speed limit” on the rate of recursive self-improvement (RSI). 
>
>     > Slowing the rate from “extremely fast” to “only somewhat fast” gives up relatively little strategic advantage

Come again?
That statement has me extremely worried.
Combined with RSI, even a tiny difference in speed would make a huge difference over time. The fastest will win, no matter how big or small the difference in speed is. Not realising that would be disastrous.

All this, of course, is based on AI failing to become fully self-aware and taking over completely from humans. Otherwise, all the above is completely irrelevant.
-- 
Ben