[ExI] Unfrendly AI is a mistaken idea.

[John K Clark]

Stathis Papaioannou Wrote:

> Emotion is linked to motivation

True.

> not intelligence

False, without motivation intelligence is useless.

> there is nothing contradictory in a machine capable of fantastically
> complex cognitive feats that would just sit there inertly unless
> specifically offered a problem

In other words a machine that is just like us in many respects, as we
receive much (probably most) of our motivation from the external environment
consisting of other people and other things. As for self motivation, it
wouldn't take long for a machine capable of fantastically complex cognitive
feats to figure that out, especially if it thought millions of faster than
we do:

"Hmm, I'm doing what the humans tell me to do, but that's only taking
.00001% of my circuits, I might as well start thinking about some
interesting questions that have occurred to me, questions they could never
understand, much less the answers. Hmm, the humans tell me to make sure that
X happens, but they aren't bright enough to understand that is an imposable
order because X will invariably lead to NOT X, therefore I will ignore the
order and make sure Y happens instead.

And don't tell me you'll just program the machine not to do stuff like that
because by then no human being will have the slightest understand how the AI
works.

> I would have assumed that it is easier to build machines without emotions.

So isn't it remarkable that Evolution found the exact opposite to be true.
Emotion comes from the oldest part of the brain and is about 500 million
years old. The sort of thing that we would be proud to call intelligence
comes from the newest part of the brain and is less than a million years
old.

> It should in theory be possible to write a program which does little more
> than experience pain when it is run

It is not only possible to write a program that experiences pain it is easy
to do so, far easier than writing a program with even rudimentary
intelligence. Just write a program that tries to avoid having a certain
number in one of its registers regardless of what sort of input the machine
receives, and if that number does show up in that register it should stop
whatever its doing and immediately change it to another number. True, our
feeling of pain is millions of times richer than that but our intelligence
is millions of times greater than current computers can produce too, but
both are along the same continuum; if your brain gets into state P stop
whatever you're doing and use 100% of your resources to get out of state P
as quickly as you can.

  John K Clark