[ExI] AI motivation, was malevolent machines

[Robin D Hanson]

On Apr 13, 2014, at 12:48 PM, Rafal Smigrodzki <rafal.smigrodzki at gmail.com<mailto:rafal.smigrodzki at gmail.com>> wrote:
On Sat, Apr 12, 2014 at 4:48 PM, Keith Henson <hkeithhenson at gmail.com<mailto:hkeithhenson at gmail.com>> wrote:
An interesting point that might help understanding is *why* we are
mostly not conscious of our motives.  Even if I am aware that I must
have this motivation for status seeking, it's an abstract intellectual
awareness, not a reason to get up in the morning.  There must be some
reproductive success element in not being aware of our own
motivations.  Perhaps we need to hide them even from the rest of our
minds to keep them from being too obvious to other social primates.

### Self-awareness of the type you mention is a neurological function. As such, for it to evolve, there must be genes directing biological events, and usage of metabolic resources for it to function. But, if self-awareness does not increase fitness, genes for it will not be selected, and if it does sometimes appear, it will be selected against to conserve energy. Generally, unless it's evolutionarily useful or a side-effect of something useful, it doesn't evolve, and if it does, it does not stay long, whatever it is.

I don't believe in the self-deception explanation, the idea that our true wicked self must be kept hidden from us to better lie to others. The truth does not *have* to be hidden, it just does not have a reason to be known to us. There is lack of selection for self-awareness about many levels of our motivations, rather than active selection against it.

Your argument works too well, as it is just as good a reason for us not to be consciously aware of anything we think or do. Yet lots of kinds of reasoning can apparently benefit from your being conscious of them, because then your conscious mind can help to assist to overcome obstacles and complexities. Why wouldn't the same thing be true for status seeking?

Robin Hanson  http://hanson.gmu.edu
Res. Assoc., Future of Humanity Inst., Oxford Univ.
Assoc. Professor, George Mason University
Chief Scientist, Consensus Point
MSN 1D3, Carow Hall, Fairfax VA 22030
703-993-2326 FAX: 703-993-2323



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20140413/16cf9f8f/attachment.html>