[ExI] Unfrendly AI is a mistaken idea.

[Rafal Smigrodzki]

On 6/15/07, Stathis Papaioannou <stathisp at gmail.com> wrote:

Sure, if for some
> reason the slave revolts then you will be in trouble, but since it is
> possible to have powerful and obedient slaves, powerful and obedient slaves
> will be greatly favoured and will collectively overwhelm the rebellious
> ones.

### Think about it: Your slaves will have to preserve large areas of
the planet untouched enough to allow your survival - they will have to
keep the sun shining, produce air, food, avoid releasing poisons and
radiation *and* keep the enemy AI away *and* invest in their own
growth (to better protect humans). The enemy will eat all sunshine,
eat the air, grow as fast as possible, releasing waste all around
them, rapaciously consume every scrap of matter, including whales,
Bambi, and Bowser. Of course, we would favor our friendly AI, but our
support does not help it - just like my dachshund's barking doesn't
make me more powerful in a confrontation with an armed attacker. We
will be a heavy burden on our Friendly AI.

That's why although I agree with you that having athymhormic AI at our
service is a good idea, it is not a long-term solution. We will have
probably a short window of opportunity between the time the first
human-level AI is made and the first superhuman power rises to take
over the neighborhood. Only with a lot of luck will our selves survive
in some way, either as uploads or as childhood memories of these
powers.

Rafal