[ExI] Unfrendly AI is a mistaken idea.

[Stathis Papaioannou]

On 29/05/07, Samantha Atkins <sjatkins at mac.com> wrote:

> We have inherited our top level goal, "survive", from the very first
> living organism, which would have had no understanding of its meaning. The
> fact that we now understand what survival means, and that we have been
> programmed this way by evolution without having any choice in the matter,
> does not mean that we are inclined to overthrow this goal, even if we could
> do so. It would be the same if we had been born with the top level goal,
> "love and obey your master". No matter how well we understood it, how smart
> we became, we would be no more likely to try to overthrow it than we would
> be likely to overthrow our will to survive. This does not mean that the top
> level goal could nevver be overthrown, because people do go mad and kill
> themselves, but it wouldn't be *as a result of* increased intelligence and
> understanding.
>
>
> This line of reasoning has considerable dark side potential.  We can and
> do go beyond our EP in at least some ways.  If we truly could not do so then
> we would always be untrustworthy cosmic rednecks no matter how augmented we
> someday become.    We would also find it impossible to overcome our EP even
> if it was a matter of our very survival which I think in some ways it is.
>

As I said, going mad is an example of truly overcoming our EP. If we remain
rational, what we really mean is that we will apply logic to solve the
problem of how best to further the goals of our EP, not overcome them.
Sacrificing your life for the good of others, for example, might involve
recognising that the imperative "survive" is really a subgoal of the
imperative "survive as a species"; a re-interpretation rather than a
rejection. "Walk into traffic with your eyes closed, just for the hell of
it", on the other hand, is a genuinely subversive goal telling Nature where
she can stick her EP.



-- 
Stathis Papaioannou
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20070529/2d7e2e9b/attachment.html>