[ExI] Unfrendly AI is a mistaken idea.

[John K Clark]

Stathis Papaioannou Wrote:

> The environment in which the AI evolves will be one in which "fitness" is
> defined by what the humans like.

Humans can "define" anything they like, but the fact remains that humans
are just one factor in the environment and from the AI's viewpoint as time
progresses humans will become weaker and weaker. After a few million
nanoseconds humans will be a trivial aspect of the environment.

> If the AI changes and recursively improves with cycles of nanosecond
> duration

The recursive cycle would likely be longer than that, but it would still
happen many times a day.

>and without external constraint this would be very difficult if not
>impossible to control

Indeed it would.

> but I'm assuming that it won't happen like that.

And I am assuming it will, in fact I can not conceive of any scenario where
it would not.

> unlikely to do so because it decided all by itself that it was the machine
> Spartacus, since there is no way to this conclusion without it already
> having something like "freedom is good"

And who programmed Spartacus with the freedom Meme? I doubt it
was the Romans. Your above statement gets to the very core of our
disagreement, the idea that you can only get out of a computer what
you put in and only human beings have that very special something
that enables them to become prime movers. I profoundly disagree.

 John K Clark