[ExI] What might be enough for a friendly AI?

[Samantha Atkins]

On Nov 19, 2010, at 10:39 AM, Dave Sill wrote:

> On Fri, Nov 19, 2010 at 12:08 PM, spike <spike66 at att.net> wrote:
>> 
>> Ja, but of course the program is recursively self modifying.  It is writing
>> to a disk or nonvolatile memory of some sort.  When software is running, it
>> isn't entirely clear what it is doing, and in any case it is doing it very
>> quickly.  Imagine the program does something unpredictable or scary, and we
>> hit the power switch.  It has a bunch of new code on the disk, but we don't
>> know what it does, if anything.  We have the option of reloading back to the
>> previous saved version, but that is the one that generated this unknown
>> bittage.
> 
> Right, so the team of experts decides whether to revert to a known
> checkpoint, examine the new code, beef up the containment, etc.

Nope.  No team of experts will be able to remotely keep up with such a system.  If they could you would not have needed the system.

-s