[ExI] Hard Takeoff

[BillK]

On Wed, Nov 17, 2010 at 4:50 PM, Richard Loosemore  wrote:
> This is why the issue of defining "friendliness" in a rigorous way is so
> important.
>
> I have spoken on many occasions of possible ways to understand this concept
> that are consistent with the way it is (probably) implemented in the human
> brain.  The basis of that approach is to get a deep understanding of what it
> means for an AGI to have "motivations".
>
> The problem, right now, is that most researchers treat AGI motivation as if
> it were just a trivial extension of goal planning.  Thus, motivation is just
> a stack of goals with an extremely abstract (super-)goal like "Be Nice To
> Humans" at the very top of the stack.  Such an idea is (as I have pointed
> out frequently) inherently unstable -- the more abstract the goal, the more
> that the actual behavior of the AGI depends on a vast network of
> interpretation mechanisms, which translate the abstract supergoal into
> concrete actions.  Those interpretation mechanisms are a completely
> non-deterministic complex system.
>
> The alternative (or rather, one alternative) is to treat motivation as a
> relaxation mechanism distributed across the entire thinking system. This has
> many ramifications, but the bottom line is that such systems can be made
> stable in the same way that thermodynamic systems can stably find states of
> minimum constraint violation.  This, in turn, means that a properly designed
> motivation system could be made far more stable (and more friendly) than the
> friendliest possible human.
>
> I am currently working on exactly these issues, as part of a larger AGI
> project.
>
>
>
> Richard Loosemore
>
>
> P.S.   It is worth noting that one of my goals when I discovered the SL4
> list in 2005 was to start a debate on these issues so we could work on this
> as a community.  The response, from the top to the bottom of the SL4
> community, with just a handful of exceptions, was a wave of the most
> blood-curdling hostility you could imagine.  To this day, there exists a
> small community of people who are sympathetic to the approach I described,
> but so far I am the only person AFAIK working actively on the technical
> implementation.  Given the importance of the problem, this seems to me to be
> quite mind-boggling.
>
> SIAI, in particular, appears completely blind to the goal-stack instability
> issue I mentioned above, and they continue to waste all their effort looking
> for mathematical fixes that might render this inherently unstable scheme
> stable.  As you saw from the deafening silence that greeted my mention of
> this issue the other day, they seem not to be interested in any discussion
> of the possible flaws in their mathematics-oriented approach to the
> friendliness problem.
>
>


That's the trouble with smart male geeks. They want everything to be
logical and mathematically exactly correct. Anything showing traces of
emotion, caring, 'humanity' is considered to be an error in the
programming.
How something can be designed to be 'Friendly' without emotions or
caring is a mystery to me.


BillK

PS
Did you know that more than one million blokes have been dumped by
their girlfriends – because of their obsession with computer games?
<http://swns.com/one-million-men-dumped-because-of-computer-game-obsessions-151620.html>