[ExI] AI motivation, was malevolent machines

Fri Apr 25 05:11:31 UTC 2014

On Sun, Apr 13, 2014 at 12:57 PM, Robin D Hanson <rhanson at gmu.edu> wrote:

>
>  On Apr 13, 2014, at 12:48 PM, Rafal Smigrodzki <
> rafal.smigrodzki at gmail.com> wrote:
>
> On Sat, Apr 12, 2014 at 4:48 PM, Keith Henson <hkeithhenson at gmail.com>wrote:
>
>> An interesting point that might help understanding is *why* we are
>> mostly not conscious of our motives.  Even if I am aware that I must
>> have this motivation for status seeking, it's an abstract intellectual
>> awareness, not a reason to get up in the morning.  There must be some
>> reproductive success element in not being aware of our own
>> motivations.  Perhaps we need to hide them even from the rest of our
>> minds to keep them from being too obvious to other social primates.
>>
>>  ### Self-awareness of the type you mention is a neurological function.
> As such, for it to evolve, there must be genes directing biological events,
> and usage of metabolic resources for it to function. But, if self-awareness
> does not increase fitness, genes for it will not be selected, and if it
> does sometimes appear, it will be selected against to conserve energy.
> Generally, unless it's evolutionarily useful or a side-effect of something
> useful, it doesn't evolve, and if it does, it does not stay long, whatever
> it is.
>
>  I don't believe in the self-deception explanation, the idea that our
> true wicked self must be kept hidden from us to better lie to others. The
> truth does not *have* to be hidden, it just does not have a reason to be
> known to us. There is lack of selection for self-awareness about many
> levels of our motivations, rather than active selection against it.
>
>
>  Your argument works too well, as it is just as good a reason for us not
> to be consciously aware of anything we think or do. Yet lots of kinds of
> reasoning can apparently benefit from your being conscious of them, because
> then your conscious mind can help to assist to overcome obstacles and
> complexities. Why wouldn't the same thing be true for status seeking?
>

### Indeed, this is an argument from biological principles applicable to
conscious awareness of any motivations. Before we go on let me mention what
I mean by "conscious" here:

To be conscious of something means to pay attention to it, hold a mental
image of it, manipulate that mental image, compare it to other images, and
(usually) be able to assign a symbol to it to differentiate it from other
images. Consciousness is highly theory-laden: There is a lot of assumptions
(theory) inherent in any conscious processing, even more than in simple
perceptions (which are not that simple either, once you analyze them in
detail). Most of our body processes run under the radar of consciousness
most of the time but many are accessible to consciousness. You may choose
to pay conscious attention to the workings of your belly. With appropriate
medical training, that is with acquisition of additional theories, you can
even mentally dissect some of the otherwise opaque signals: For a surgeon,
a colicky pain in the right hypochondriac region occurring after fatty
meals warns of gallstones but a similar pain the right illiac region with
rebound warns of appendicitis, potentially much more dangerous. We are
evolved with the neural hardware we can use to make advantageous decisions
under natural conditions but not much more. Untrained, we can feel a
bellyache, and we don't need to know all the details to take the hint and
avoid fights and hunting until it goes away. The finer signal analysis
options accessible to a surgeon were useless until the invention of
surgery, and therefore we have not evolved to make them. The belly does not
tell us the truth, yet we don't say it deceives us.

Now, motivation is a body process as well, although its moving parts are
inside our brain, not in our bellies. Are we evolutionarily incentivized to
be fully conscious of the minutiae of motivations in general? Do we need to
know the evolutionary influence behind our motivations?I would say not, at
least not under conditions of evolutionary adaptiveness. We are motivated
to e.g. seek fatty food, healthy-appearing mates and to seek social status.
These are valid, effective proxies (signal sources) for evolutionary
success - directly or indirectly leading to the survival of our individual
genes. To be successful in the ancestral veldt we needed to consciously
analyze the ways of achieving these goals but we did not need to understand
their evolutionary roots or their neural underpinnings. Nowadays, the
situation is different - to avoid the evolutionary traps of e.g. female
careerism, pornography or obesity, it pays in fitness terms to be able to
analyze the situation consciously and make fitness-enhancing choices. But,
judging from falling birthrates and ballooning waistlines, this sort of
self-awareness is not too common.

To say that there is a species-wide mechanism of self-deception that
actively hides status seeking from our inner eye would be to posit the
following:

- the ability to consciously analyze status evolved, possibly 150 000 years
ago (implying there is a fitness premium for conscious analysis of status,
over and above of what get from instinctive following of status
signals/proxies)
- subsequently there was massive evolutionary pressure to hide this ability
- an effective pan-species neural mechanism evolved to maintain the
fitness-enhancing effect of conscious analysis while hiding it effectively
from introspection

I find this scenario implausible. Humans are clearly involved in an
evolutionary arms race between liars and liar-spotters, between
dominance-seekers and the envious or fearful puller-downs. But did we
evolve to detect status seeking in general or just the dangerous,
machiavellian, back-stabbing type? There is research showing that most
humans can subliminally detect psychopaths. But most of allow status
achieved through licit means. We do not automatically turn on all
high-status individuals, we seek to affiliate with some of them. We try to
detect and isolate cheats but we revere heroes.

There is also interesting research on high-status liars: High status people
apparently lie much better to low-status ones, while low-status ones have
difficulties lying to high-status ones. Being embarrassed about lying helps
avoid lying dangerously, e.g. when talking to a chief who can kill you. But
why self-deceive when deceiving a weak person?

There is space here for various secondary and tertiary effects, and complex
evolutionary equilibria. The simple idea that it pays to be unaware of
status-enhancing effects of your actions can neither be derived from first
principles nor is it to the best of my knowledge directly proven
observationally. The self-deception hypothesis can be an inspiration for a
research program which might first have to delve into more basic questions,
like "What is the evolutionary mechanism of status differences in primate
groups?", "What specific neural mechanisms underlie status detection and
how do they differ between humans and lower primates?", "Do other species
self-deceive?"

There is woefully too little data to accept the self-deception hypothesis
at this time.

Rafal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20140425/6a43a2c6/attachment.html>