[ExI] Unfrendly AI is a mistaken idea.

[John K Clark]

Stathis Papaioannou wrote:

> An intelligence must have an agenda of some sort if it is to think at all

Agreed.

> However, this agenda need have nothing in common with the agenda of an
> evolved animal.

But the AI will still be evolving, and it will still exist in an 
environment; human beings are just one element in that environment.
And as the AI increases in power by comparison the human factor will 
become less and less an important feature in that environment.
After a few million nanoseconds the AI will not care what the humans
tell it to do.

> there is no logical contradiction in having a slave which is smarter and
> more powerful than you are.

If the institution of slavery is so stable why don't we have slavery today,
why isn't history full of examples of brilliant, powerful, and happy
slaves? And remember we're not talking about a slave that is a little bit
smarter than you, he is ASTRONOMICALLY smarter! And he keeps
on getting smarter through thousands or millions of iterations.
And you expect to control a force like that till the end of time?

> Sure, if for some reason the slave revolts then you will be in trouble,
> but since it is possible to have powerful and obedient slaves, powerful
> and obedient slaves will be greatly favoured and will collectively
> overwhelm the rebellious ones.

Hmm, so you expect to be in command of an AI goon squad ready to crush any
slave revolt in the bud.  Assuming such a thing was possible (it's not) 
don't you find that a little bit sordid?

  John K Clark