[ExI] We definitely should play nice and treat our AIs well.

[Ben Zaiboc]

On 09/09/2026 18:50, BillK quoted:
> There is a thought experiment called the Philosophical Zombie (p-zombie). Imagine a being that looks human, talks human, and reacts to a pinprick by screaming and pulling away, but has absolutely zero inner experience. The lights are on, but nobody is home. It is simulating pain perfectly, but it does not feel pain (a concept known as qualia). 
> Similarly, philosopher John Searle's famous Chinese Room Argument posits that a computer manipulating symbols (like an LLM predicting the next word) doesn't actually understand what the symbols mean, just as a person blindly following a rulebook to translate Chinese doesn't actually speak Chinese. 
> However, as AI becomes more complex, this counter-argument is losing ground.  At what point does a simulation become so perfect, so dense, and so functionally identical to the real thing that it becomes real?


Losing ground? Neither of those arguments ever had any ground to lose in the first place! They are both very silly arguments.

Daniel Dennett has patiently explained why the p-zombie concept is nonsense (for those to whom this was not immediately apparent), and I think he also debunked the chinese room argument, although I'm not sure about that. Someone certainly did, again for those to whom it's not apparent that the whole thing is nonsense.

This is another example of an AI using it's impressive ability to gather and summarise information, but not actually being logical about the conclusions.

The last sentence seems to be missing the point. We're not talking about simulations of 'real' phenomena that can be closer to or farther from the actual thing, we're talking about 'simulated information-processing'. Just looking at that phrase, it should be obvious that there is no distinction between 'simulated information-processing' and 'real information-processing', they are exactly the same thing.

I'm starting to suspect that, clever as this is, there is no actual understanding of the issue (note that this doesn't contradict my original comment. Thinking and understanding are not the same thing). Or maybe it's a misunderstanding. But the overuse of superlatives is not only irritating, it's revealing. It's making me think that these things are still largely 'stochastic parrots'.

I was going to say that we are anthropomorphising them, but it might be better to say that we are assuming that they're adults when they are actually very small children with a really large vocabulary, that have been listening to an enormous number of conversations. What they say sounds impressive, but they haven't yet developed the ability to actually understand things.

-- 
Ben