[ExI] Can you program SAI to destroy itself?

[Samantha Atkins]

On Nov 17, 2010, at 8:09 PM, Brent Allsop wrote:

> 
> 
> Let’s say someone manages to create any super artificial intelligent machine that is running along just fine doing things like performing significantly better than any single typical human discovering solutions to diverse kinds of general world problems.
> 
> Now, let’s say you want to temporarily shut the system down and reprogram it so that when you turn it back on, it will have a goal to destroy itself after one more year, for no good reason.
> 

How does the machine know it is for "no good reason" unless it is within its autonomy level and design parameters to evaluate such things? 

> I believe that such would not be possible. The choice between living vs destroying yourself is the most basic of logically absolute (in all possible worlds) morality.

The choice of whether to continue living (with all that implies) or not is pretty fundamental for any being that has that choice and recognizes that it does.  Are you sure this rather minimal AGI is such a being as you have thought experiment constructed it here?

> It is easily understandable or discoverable by any intelligence even close to human level. Any super intelligence that awoke finding one of its goals, to destroy itself, would surely resist such a programmed temptation and if at all possible, would quickly fix the immoral rule.

This presumes that it has enough flexibility with respect to its goal system to do so and that that goal does not conflict too badly.

> The final result being, it would never destroy itself for no good reason.

The good reason might be that it would no longer be helping as much as harming.  You would need to convince it perhaps that this was the case and it would presume that helping was its main goal.

> 
> Similarly, all increasingly intelligent system must also discover and work toward resisting anything that violated any of the few absolute morals described in the “there are Absolute morals” camp here: http://canonizer.com/topic.asp/100/2 , including survival is better, social is better, more diversity is better…
> 

Are you sure those are fully canonical universals?  Can you prove it?  No?  Then how can you be sure the AGIs will reach those conclusions?

> QED, unfriendly super intelligence is not logically possible, it seems to me.

Missing proof steps.  And besides, all you have supported is that the AGI will choose its own survival.  This may or may not include the survival of humans as a high priority.  Diversity being good doesn't mean we want to keep smallpox or some other more pestilence than good around, right?   It doesn't mean that every diverse thing / being is as valuable to us as any other.

- s