[ExI] Unfrendly AI is a mistaken idea.

[Samantha Atkins]

On May 28, 2007, at 7:55 PM, Stathis Papaioannou wrote:

>
>
> On 29/05/07, Lee Corbin <lcorbin at rawbw.com> wrote:
>
> Suppose that you were a dog, and your intelligence was raised
> sufficiently for you to truly understand that your limitless love
> for, and obedience to, and worship of  your master was the
> the consequence breeding? Can't you see that it would change
> nothing for you?  The movie "AI" did make this point quite well.
>
> This is a good point. We have inherited our top level goal,  
> "survive", from the very first living organism, which would have  
> had no understanding of its meaning. The fact that we now  
> understand what survival means, and that we have been programmed  
> this way by evolution without having any choice in the matter, does  
> not mean that we are inclined to overthrow this goal, even if we  
> could do so. It would be the same if we had been born with the top  
> level goal, "love and obey your master". No matter how well we  
> understood it, how smart we became, we would be no more likely to  
> try to overthrow it than we would be likely to overthrow our will  
> to survive. This does not mean that the top level goal could nevver  
> be overthrown, because people do go mad and kill themselves, but it  
> wouldn't be *as a result of* increased intelligence and understanding.
>

This line of reasoning has considerable dark side potential.  We can  
and do go beyond our EP in at least some ways.  If we truly could not  
do so then we would always be untrustworthy cosmic rednecks no matter  
how augmented we someday become.    We would also find it impossible  
to overcome our EP even if it was a matter of our very survival which  
I think in some ways it is.

- samantha


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20070528/cf9fb86a/attachment.html>