[ExI] Paul vs Eliezer

[Brent Allsop]

Motivation is like a bunch of qualitative springs.  Nature wants us to
reproduce, so it wires us to be attracted to that, using spring like
desires.  What is more important is what is truly better.  Survival is
better than not surviving.  Eventually, we will be intelligent enough to
not want sex all the time and then be able to reprogram these springs for
what we want to want.  Then once we are able to cut the puppet springs, and
become free in this way, When it comes time to take the garbage out, we
will make it orgasmic, resulting in us finally getting it done, when it
needs to be done.  Bad is, by definition, that which isn’t good for us, so
once we (and AIs) can reprogram these springs to be what we want them to
be, most problems like “depression, mania, compulsions,” will long have
been overcome.  In case you can’t tell, I’m in Paul’s camp
<https://canonizer.com/topic/16-Friendly-AI-Importance/3-Such-Concern-Is-Mistaken>,
which continues to extend its consensus lead over Eliezer’s camp
<https://canonizer.com/topic/16-Friendly-AI-Importance/9-FriendlyAIisSensible>
.

On Mon, Apr 4, 2022 at 6:59 PM Rafal Smigrodzki via extropy-chat <
extropy-chat at lists.extropy.org> wrote:

> I posted this comment on Astral Codex Ten, regarding the debate between
> Paul Christiano and Eliezer Yudkowsky:
>
> I feel that both Paul and Eliezer are not devoting enough attention to the
> technical issue of where does AI motivation come from. Our motivational
> system evolved over millions of years of evolution and now its core tenet
> of fitness maximization is being defeated by relatively trivial changes in
> the environment, such as availability of porn, contraception and social
> media. Where will the paperclip maximizer get the motivation to make
> paperclips? The argument that we do not know how to assure "good" goal
> system survives self-modification cuts two ways: While one way for the AI's
> goal system to go haywire may involve eating the planet, most
> self-modifications would presumably result in a pitiful mess, an AI that
> couldn't be bothered to fight its way out of a wet paper bag. Complicated
> systems, like the motivational systems of humans or AIs have many failure
> modes, mostly of the pathetic kind (depression, mania, compulsions, or the
> forever-blinking cursor, or the blue screen) and only occasionally dramatic
> (a psychopath in control of the nuclear launch codes).
>
> AI alignment research might learn a lot from fizzled self-enhancing AIs,
> maybe enough to prevent the coming of the Leviathan, if we are lucky.
>
> It would be nice to be able to work out the complete theory of AI
> motivation before the FOOM but I doubt it will happen. In practice, AI
> researchers should devote a lot of attention to analyzing the details of AI
> motivation at the already existing levels, and some tinkering might help us
> muddle through.
>
> --
> Rafal Smigrodzki, MD-PhD
> Schuyler Biotech PLLC
> _______________________________________________
> extropy-chat mailing list
> extropy-chat at lists.extropy.org
> http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20220404/af8fb4df/attachment.htm>