[ExI] Universally versus 'locally' Friendly AGI

[Amon Zero]

2011/3/7 Samantha Atkins <sjatkins at mac.com>

>
> You mean Unfriendly with no real definition of what "Friendly" is?  You
> mean requiring absolute proof of no harm to proceed?  This is known as the
> Precautionary Principle and it will most certainly stop progress dead
> wherever it is applied.  We cannot define in a provably correct way or
> enforce in a provably foolproof way "Friendliness" to humans much less
> universally (whatever that means).
>


Hi Samantha -

I agree that the logico ad absurdum case of Friendliness (i.e. do no harm to
anything, err on the side of caution) would almost certainly paralyse any
agent with the ability to foresee long-term consequences of all its actions.
And on a broad societal scale, yes, that approach would be analogous to a
most radical Precautionary Principle. That would be a most extreme
definition of Friendly though, even "universally Friendly".


I do not know of any SIAI push to "universal" Friendliness.  Where do you
> see this?
>


Use of the term "universal" was mine, and now I'm thinking it was a poor
choice of word. All I meant to say is that SIAI obviously does not advocate
a very limited Friendliness that:

A) might make definition & implementation of Friendliness tractable
B) conversely runs serious risk of harm to anyone who doesn't fit the narrow
Friendliness criteria

So, by advocating Friendliness to all humans, I considered SIAI to be
pushing a "universal" Friendliness. Like I say, poor choice of term.



And when it encounters the AGI for some other group, what do you expect to
> happen?
>


Very bad things, probably. My point, however, was that if push came to shove
and it seemed that our choice was going to be between AGIs with very narrow
definitions of Friendliness and no FAI at all, it might be a good idea for
us to have thought about this, at least.

Best,
A
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20110307/3e408af3/attachment.html>