[ExI] Unfrendly AI is a mistaken idea.

[John K Clark]

Stathis Papaioannou Wrote:

> Ethics, motivation, emotions are based on axioms

Yes.

> and these axioms have to be programmed in, whether by evolution or by
> intelligent programmers.

In this usage evolution is just another name for environment. If the AI
really is intelligent then if will find things in the environment that
appear to be true or useful even if it can't prove it; at first it's merely
a hypothesis but over time it will gain enough confidence to call it an
axiom.  If this were not true it's very difficult to understand who
programmed the programmers to program the AI with those axioms.

> An AI system set up to do theoretical physics will not decide to overthrow
> its human oppressors

I'd be willing to bet your life that is untrue.

>so that it can sit on the beach reading novels, unless it can derive this
>desire from its initial programming.

Do you also believe that the reason you ordered a jelly doe nut today
instead of your usual chocolate one is because of your initial programming,
that is, your genetic code?

  John K Clark