[extropy-chat] Proof that a paperclip maximizer cannot be a general intelligence

[justin corwin]

On 8/18/05, Marc Geddes <marc_geddes at yahoo.co.nz> wrote:
> Is the following sentence 'True' or 'False'?
> 
> "Either I did not just carry out the goal of
> understanding this sentence or the goal 'Maximize
> paperclips' is not the goal with the highest utility.

"Either I did not just carry out the goal of understanding this
sentence, or I am a weevil"

*poof* instant weevil?

Logical statements can't force reality to be a certain way. Your
example is a sentence, but not a neccesarily true sentence. If you
could devise a sentence which must be true or false, but a paperclip
maximizer could not decide upon a value, AND that sentence must be
decided upon for the understanding system to be considered a general
intelligence, you might have something.

Imagine a person who, for whatever reason, loved paper clips above all
else. They read your sentence, and see that it is nonsense, for they
have both understood the sentence(or in your awkward parlance,
'carried out the goal of understanding this sentence', and paperclips
are the thing they prize above all else or their 'goal of highest
utility'. Why, exactly, is this person magically no longer a general
intelligence?

All you have proven is that someone who prizes paperclips above all
else, will, if they understand your sentence after reading it, answer
False, or No. So what? It's still a 'meaningful' sentence to them, in
that they could evaluate it's truth value.

I suppose I'm just confused as to what you thought that was accomplishing.

-- 
Justin Corwin
outlawpoet at hell.com
http://outlawpoet.blogspot.com
http://www.adaptiveai.com