[ExI] Echoes of the Invincible

[Rafal Smigrodzki]

On Mon, Jun 24, 2013 at 5:44 AM, Eugen Leitl <eugen at leitl.org> wrote:

> But it doesn't prevent the rest of us from using darwinian
> systems which are nonbrittle, and hence antifragile.
>
> So not only does this prevent the scenario, the brittle
> strain self-terminates over time and space.

### You are probably but not guaranteed to be right. Yes, a Darwinian
system (many independent replicators able to fundamentally change
their characteristics by mutation or directed mutagenesis) is likely
to outcompete a non-evolving system (replicators with predominantly
pre-defined unchangeable characteristics that limit the repertoire of
their responses to threats), as long as both start out at a rough
parity. But, there is a situation where the rigid system can stably
take over: If the Darwinian system consists of microbes which have a
very constrained space of development paths, for reasons of simple
physics, it has to through a succession of developmental steps which
are vulnerable to brute force attacks, before accumulating enough
complexity to use directed mutagenesis. And without advanced evolution
(self-modifying AI, directed evolution) it's probably hard to take on
a superhuman AI. The microbe has to become multicellular, grow a
brain, get bigger, learn, and during this time you can destroy or
reset it down to microbe level. All it takes is an asteroid
judiciously planted every few hundred million years. So, and
established non-evolving advanced enough to survey and kill any
upstarts could remain stable till the suns burn out. Stanislaw Lem
tackled this in "The Invincible" but came to a different conclusion.

Imagine we actually are the firstborn. An AI is created with a stable
goal system, takes over the world, possibly eradicating us. The AI is
able to make non-mutating replicating versions of itself, capable of
colonizing space at 0.99 c, and completely takes over every scrap of
real estate where competing life can emerge (presumably only a subset
of planets). Local microbes have no chance of ever evolving high
enough levels of complexity to take on the AI. They can't even evolve
to infect the AI's bodies - I am assuming that an AI could recompile
(for lack of a better word) its physical implementation to always
remain invulnerable to a crude replicator attack. The AI could switch
between trillions of informatically equivalent but chemically
completely different body designs, and a microbial biosphere would
never have a chance of getting a foothold, since a microbe always
recognizes a small number of chemical characteristics of a target (a
number limited by the size of a microbial genome).

The intellectual history of the visible universe ends right there.

Rafal