[ExI] DARPA killer AI robots to 'participate in own construction'

[Dave Sill]

2009/6/2 Stefano Vaj <stefano.vaj at gmail.com>:
>
> You've got your robots which can make copies of themselves, of course.
> That's pretty scary - a runaway exponentially-multiplying machine horde,
> potentially able to overrun the human race in an eyeblink. But how much more
> scary would it be if you had a machine which could not only make copies of
> itself once complete, but could also participate in its own construction
> while it was still being built?

Not at all. I don't see the threat.

> DARPA seeks to construct systems that can participate in their own
> construction... The system might know the requirements for various tasks in
> its repertoire, and it may try to perform those tasks to verify
> functionality.

Key word being "may".

> "Tasks in its repertoire", then, might include "destroy all moving objects
> within 100 miles not designated as 'friendly'" or "mount an immediate
> armoured assault on Beijing, regardless of nuclear response". The prospect
> of the software unilaterally "trying to perform those tasks to verify
> functionality" doesn't seem reassuring.

The people designing these things aren't idiots, unlike some alarmist
reporters who've watched one too many Terminator movie.

> In any case, it seems plain that building a system focused on "high-level
> cognition" which can "participate in its own construction" will be fraught
> with difficulty. You might have a notion of what you'd like it to be, but it
> will have its own ideas. By definition, one would have no firm picture of
> just what would be unleashed upon an unsuspecting world at the end of the
> construction process - but you would know that it'd be potentially able to
> make more of itself, or indeed repair itself if it got damaged.

No. "Construction" is not "design". These aren't intelligent robots
designing themselves, they're appliances that can help bootstrap their
construction.

-Dave