[ExI] Unfrendly AI is a mistaken idea.

[Lee Corbin]

In an otherwise reasonable post, John Clark writes

> It is not only possible to write a program that experiences pain it is easy
> to do so, far easier than writing a program with even rudimentary
> intelligence. Just write a program that tries to avoid having a certain
> number in one of its registers regardless of what sort of input the machine
> receives, and if that number does show up in that register it should stop
> whatever its doing and immediately change it to another number.

Any behavior of any creature whatsoever that is this simple
does not deserve to be called pain. That's the same error
that you were criticizing, namely, to call a three line program
"intelligent" in any sense. 

Pain involves at least (i) a consideration of how an entity might
extricate itself from the painful situation (ii) laying down memories 
of the steps leading to the current predicament so as to cause the
entity to avoid the predicament in the future (iii) invocation of
unpleasant emotion, such as fear, anger, or dread.

Like other complex behaviors, the capacity for pain---totally
absent in plants---took millions of years to evolve. It should
be looked at as a highly complex and evolved behavior.

Lee