[ExI] Echoes of the Invincible

[Rafal Smigrodzki]

On Tue, Jun 25, 2013 at 4:11 AM, Eugen Leitl <eugen at leitl.org> wrote:
> On Tue, Jun 25, 2013 at 12:19:40AM -0400, Rafal Smigrodzki wrote:
>
>> Imagine we actually are the firstborn. An AI is created with a stable
>> goal system, takes over the world, possibly eradicating us. The AI is
>
> You just described something which can't be built by us, a
> Darwinian system. Have you noticed how explicit-codifying
> AI never went anywhere, and most people continued to plod
> along, as if nothing happened? The task is much too hard
> for anything less than a god (which can have arisen by
> a darwinian design, and could plunk down a hyperfit brittle
> system down in our midst which would flatten us, but be
> a pitiful toy in the original context it arose in, which
> *will* be expansive, so no chances for that hyperfit
> oddness cropping up in our midst.
>
> This is what I mean that there is no mechanism. People
> armwave a lot, but that's unfortunately not enough.

### I am assuming you expect that superhuman AI will be produced, just
not using explicit programming techniques.

Leaving aside the question of that programming techniques are employed
in creating it, do you disagree that a superhuman AI with a stable
goal system could exist? Maybe it could only be evolved using
evolutionary programming techniques, maybe something more directly
controlled by programmers but an  AI with a stable goal system, and
capable of cooperating with its copies, would create a stably
non-evolving society, as long as the AI were smart enough to suppress
the emergence of competing, evolving replicators.

I agree that the stable-goal SAI is likely to perish if pitted against
an ecosystem of less stable, evolving SAIs but this is not the
situation I am considering here.

Please note, the reason for the seemingly progressive nature of
evolution is a combination of lack of a goal system of the process as
a whole, and the existence of progressively more complex ecological
niches that can be reached by progressively more complex beings. As
long as something can evolve (i.e. there is an ecological niche for
it), it will evolve, since there is no designer with god's-eye view of
the process to stop it at some pre-determined point.

The ecology dominated by SAI with stable goals would be different - if
the SAI's goals involved prevention of competitor evolution, the SAI
could nip the competition in the bud, before it became intelligent
enough to pose too much of a challenge. The stable SAIs could keep
eradicating upstart microbes forever, without ever having to deal with
a real opponent.

----------------

> My scenario is exactly the same, except that there is no AI,
> the first wavefront organisms are low-diversity and steamroll
> across pristine stellar terrain and pre-expansive observers
> alike, and subsequent organisim waves pass by adding complexity
> and completely displacing pioneers, until a semi-steady simmering
> state results, where the local ecosystem varies so widely
> across relatively small compartments so that crossing over
> species don't have an edge either way, and the result is
> entirely independant from the nucleating point that begat
> that particular ecosystem. Which means that when these
> meet, nothing exciting happens.

### Indeed, this is the other possibility and I agree with you it's
more likely than the non-evolving history - but which one will be
realized is highly dependent on the starting condition.

Which could occur in the next 100 years on this planet.

---------

Your brittle system
> is at a disadvantage. The only way it could meta-stable is a population
> of brittle monoclones, which will be wiped out completely if
> it encounters an advanced darwinian system.

### Agreed.

Rafal