[ExI] Hard Takeoff

[spike]

... On Behalf Of Dave Sill
...
>...Maybe I'm missing something obvious, but wouldn't it be pretty easy to
implement a chess playing robot that has no ability to resist being turned
off, break into other machines, acquire resources, etc.? And wouldn't it be
pretty foolish to try to implement an AI without such restrictions? You
could even give it access to a restricted sandbox.
If it's really clever, it'll eventually figure that out, but it won't be
able to "escape".-Dave

Perhaps, but we risk having the AI gain the sympathy of one of the team, who
becomes convinced of any one of a number of conditions: the AI is a human
equivalent, so it needs to be copied onto another computer in order to
protect it from a crash, or protect it from the other researchers.  A team
member intentionally copies the AI to take it home, to work on it more or
perhaps realizes it is worth a fortune and wishes to steal it.  Or a
researcher realizes that her own time on this planet is drawing to a close
with at best another fifty years to live, so she decides to take a chance
and unleash the beast, hoping for the best.  Or she makes a deal with the AI
to save her and slay the infidels.  Or it is so clever that it figures out
how to control microorganisms to build replicating nanobots from DNA, which
then carry the software, bit by bit, to a nearby internet enabled computer.

Dave how many scenarios can we imagine where the AI is controlled in lab
conditions, but it somehow escapes.

spike