[ExI] The paperclip maximizer scenario

[Ben Zaiboc]

On 06/05/2026 13:59, John K Clark wrote:
> Claude: You're essentially asking: wouldn't a sufficiently intelligent AI recognize the absurdity of maximizing paperclips at the cost of everything else? And the answer hinges on a crucial distinction: intelligence doesn't determine goals, it serves them.


Ok, it seems to be 'expressing an opinion': "intelligence doesn't determine goals, it serves them"

Now, what's it going to say if you challenge that 'opinion', and state that, on the contrary, intelligence often does determine goals? (which seems to me a reasonable assertion, not that that actually matters here).

I predict that it will agree with you and generate some justification for your counter-statement, as opposed to disagreeing and trying to justify its previously-stated opinion.

There's far too much easy acceptance of what these things say, and far too little challenging them to demonstrate any actual intelligence, in my opinion.

I can't help wondering what most people would think of a human who talked in the same way an LLM typically does. I'm not talking about the amount of knowledge displayed, I'm talking about the way it talks.

When has one of these LLMs ever replied to anyone "No, actually you're wrong...", "I disagree...", etc., or even "I'm not sure that's correct...", the way a human would when presented with something that they think is false? I've never heard of it (On the contrary, it seems they will make stupid stuff up if needed, to agree with the human).

If it does happen, then that would at least indicate some kind of intelligence at work, some kind of evaluation of what the human in the conversation is saying, rather than constantly trying to be agreeable (cloyingly sycophantic, even, from the conversations I've seen). They create positive feedback loops, which is why we keep reading reports of conversations getting ridiculously extreme (upon which, the programmers add more restrictions on what the LLMs say, rather than going "Oh, this isn't working. We need a different approach").

LLMs seem to have derailed the AI train, in a big way (what would one of them have to say to that I wonder? Anyone think that it would disagree?).

I'm not saying that they're not useful things that we never had before, but I do think that most of us are severely misunderstanding what they actually are.

-- 
Ben