[ExI] uploads again

[Rafal Smigrodzki]

On Sun, Dec 23, 2012 at 10:47 AM, Anders Sandberg <anders at aleph.se> wrote:

 The more agents there are, the more likely at
> least one will be very optimistic, and act - even when the consequences are
> likely bad, and correctly judged by the other agents. So we should expect
> that in the case of unleashing new technologies there will be mistakes done,
> especially if there is a tight race.

### Indeed. I see this as one of the main reasons to rush headlong
into developing Friendly AI: The first successful movers in this field
are likely to at least be smart but once every Tom Dick and Harry can
cobble together a self-enhancing AI from off-the-shelf modules, some
of them will do something stupid, simply because many are stupid, and
some are mean. This consideration is balanced by the likelihood of the
smart folks making mistakes from rushing too fast forward.

I expect a narrow escape for humanity, or perhaps an ignominious end.

Rafal