[ExI] What might be enough for a friendly AI?

[spike]

... On Behalf Of Mike Dougherty
...

On Thu, Nov 18, 2010 at 12:52 PM, Dave Sill <sparge at gmail.com> wrote:
> On Thu, Nov 18, 2010 at 12:12 PM, spike <spike66 at att.net> wrote:
>>
>> That's it Stefano, you're going on the dangerous-AGI-team-member 
>> list.  It already has Florent, Samantha, me, now you...

>I'd like to preemptively put the AGI itself on the list...

Too late, it's already there.

>...Maybe we could build two of them and make them watchdog each other...

Exactly.  Or rather, we make the first one with a team that is at least
trying to make it people-friendly, with at least some vague clue as to how
to do that.  Then the first one watches to make sure there is no second one,
like the first emergent queen bee slays her rivals.

>...Obviously they'll have incentive to outwit the other - ...

Mike you have hit upon something that has been weighing on my mind since I
realized it a couple months ago:  imagine a good-outcome endgame: an MBrain
consisting of sun-orbiting computronium, the technogeek version of heaven,
everything turned out right, all humans were voluntarily uploaded and so
forth.  But we are not finished with war!  It isn't the kind of war where
there is any injury or serious death, no projectiles, no homes burned, no
hungry refugees.  But it will still have the potential of memetic warfare, a
risk which does not necessarily diminish with time.

>...Now we just have to remember to build the off switch first and put it in
a place that we can use easily...

It isn't that simple Mike.  To use that off switch might be considered
murder.  There may not be unanimous consent to use it.  There might be
emphatic resistance on the part of some team members to using it.  It might
not be clear who is authorized to use it.  Think it over and come back
tomorrow with a list of reasons why it really isn't as simple as having a
big power cutting panic button.

spike