[ExI] What might be enough for a friendly AI?

[Mike Dougherty]

On Fri, Nov 19, 2010 at 2:07 AM, spike <spike66 at att.net> wrote:
> Mike you have hit upon something that has been weighing on my mind since I
> realized it a couple months ago:  imagine a good-outcome endgame: an MBrain
> consisting of sun-orbiting computronium, the technogeek version of heaven,
> everything turned out right, all humans were voluntarily uploaded and so
> forth.  But we are not finished with war!  It isn't the kind of war where
> there is any injury or serious death, no projectiles, no homes burned, no
> hungry refugees.  But it will still have the potential of memetic warfare, a
> risk which does not necessarily diminish with time.

We are still a threat to ourselves.  Even with an inconceivable level
of AI beneficence, we still have our nature to threaten survival.  If
that technogeek heaven is a million times more wonderful than Now, we
had better plan on dealing with manmade threats that are at least a
million times as threatening.

>>...Now we just have to remember to build the off switch first and put it in
> a place that we can use easily...
>
> It isn't that simple Mike.  To use that off switch might be considered
> murder.  There may not be unanimous consent to use it.  There might be
> emphatic resistance on the part of some team members to using it.  It might
> not be clear who is authorized to use it.  Think it over and come back
> tomorrow with a list of reasons why it really isn't as simple as having a
> big power cutting panic button.

Of course not.  Murder?  We might call it self defense.  We might call
it a lot of things; if only to justify our action.  I'm sure the ants
you tolerate for your amusement would not be so welcomed chewing into
the woodwork of your house - of course there are some who would
consider it murder for you to defend your home with poisons.
Conversely we are the ants compared to the runaway fast takeoff by the
time we realize we want to stop it - and it could conceivably have as
little concern for ridding the local environment using the most
effective pesticides available.

Perhaps I'm delusionally naive but I imagine we will continue to
evolve along with our creations so what we perceive today as a threat
to tomorrow is by that time only a compelling challenge.  We'll either
meet that challenge or we won't.  Either way it'll be a team effort.