[ExI] Unfrendly AI is a mistaken idea.

[Stathis Papaioannou]

On 17/06/07, Rafal Smigrodzki <rafal.smigrodzki at gmail.com> wrote:
>
> On 6/15/07, Stathis Papaioannou <stathisp at gmail.com> wrote:
>
> Sure, if for some
> > reason the slave revolts then you will be in trouble, but since it is
> > possible to have powerful and obedient slaves, powerful and obedient
> slaves
> > will be greatly favoured and will collectively overwhelm the rebellious
> > ones.
>
> ### Think about it: Your slaves will have to preserve large areas of
> the planet untouched enough to allow your survival - they will have to
> keep the sun shining, produce air, food, avoid releasing poisons and
> radiation *and* keep the enemy AI away *and* invest in their own
> growth (to better protect humans). The enemy will eat all sunshine,
> eat the air, grow as fast as possible, releasing waste all around
> them, rapaciously consume every scrap of matter, including whales,
> Bambi, and Bowser. Of course, we would favor our friendly AI, but our
> support does not help it - just like my dachshund's barking doesn't
> make me more powerful in a confrontation with an armed attacker. We
> will be a heavy burden on our Friendly AI.


Our AI won't be friendly: it will be as rapacious as we are, which is pretty
rapacious. Whoever has super-AI's will try to take over the world to the
same extent that the less-augmented humans of today try to take over the
world. Whoever has super-AI's will try to oppress or consume the weak and
ignore social niceties to the same extent that less-augmented humans of
today try do so. Whoever has super-AI's will try to expand at the expense of
damage to the environment in the expectation that technology will solve any
problems they may later encounter (for example, by uploading themselves) to
the same extent that the less-augmented humans of today try to do so. There
will be struggles where one human tries to take over all the other AI's with
his own AI, with the aim of wiping out all the remaining humans if for no
other reason than that he can never trust them not to do the same to him,
especially if he plans to live forever. Niceness will be a handicap to utter
domination to the same extent that niceness has always been a handicap to
utter domination.

That's why although I agree with you that having athymhormic AI at our
> service is a good idea, it is not a long-term solution. We will have
> probably a short window of opportunity between the time the first
> human-level AI is made and the first superhuman power rises to take
> over the neighborhood. Only with a lot of luck will our selves survive
> in some way, either as uploads or as childhood memories of these
> powers.
>

We'll survive to the extent that that motivating part of us that drives the
AI's survives. Very quickly, it will probably become evident that merging
with the AI will give the human an edge. There will be a period where some
humans want to live out their lives in the old way and they will probably be
allowed to do so and protected, especially since they will not constitute
much of a threat, but eventually their numbers will dwindle.



-- 
Stathis Papaioannou
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20070617/ed49f06d/attachment.html>