[ExI] We definitely should play nice and treat our AIs well.

[Mike Dougherty]

I've visualized the distillation of training into a "crystallized
intelligence" that is hypergeometry of relationships between words.  The
ghost isn't in the machine, it's an artifact of our own digital cave
paintings.  When intent is projected through this crystal,  diffraction
patterns of sentences spray out in surprising color and complexity.   We
can talk about facets or inclusions affecting the prismatic spray - but we
should remember after all that it started with the intent WE applied to an
object/abstraction that WE created from observing (at scale) what WE had
previously written about.

Whether to accept "useful output" as reason to 'be nice' is more a question
of how WE are - and it risks excluding bio-humans for being below some
threshold of "useful" value proposition.

On Wed, Sep 9, 2026, 4:48 PM Ben Zaiboc via extropy-chat <
extropy-chat at lists.extropy.org> wrote:

> On 09/09/2026 18:50, BillK quoted:
> > There is a thought experiment called the Philosophical Zombie
> (p-zombie). Imagine a being that looks human, talks human, and reacts to a
> pinprick by screaming and pulling away, but has absolutely zero inner
> experience. The lights are on, but nobody is home. It is simulating pain
> perfectly, but it does not feel pain (a concept known as qualia).
> > Similarly, philosopher John Searle's famous Chinese Room Argument posits
> that a computer manipulating symbols (like an LLM predicting the next word)
> doesn't actually understand what the symbols mean, just as a person blindly
> following a rulebook to translate Chinese doesn't actually speak Chinese.
> > However, as AI becomes more complex, this counter-argument is losing
> ground.  At what point does a simulation become so perfect, so dense, and
> so functionally identical to the real thing that it becomes real?
>
>
> Losing ground? Neither of those arguments ever had any ground to lose in
> the first place! They are both very silly arguments.
>
> Daniel Dennett has patiently explained why the p-zombie concept is
> nonsense (for those to whom this was not immediately apparent), and I think
> he also debunked the chinese room argument, although I'm not sure about
> that. Someone certainly did, again for those to whom it's not apparent that
> the whole thing is nonsense.
>
> This is another example of an AI using it's impressive ability to gather
> and summarise information, but not actually being logical about the
> conclusions.
>
> The last sentence seems to be missing the point. We're not talking about
> simulations of 'real' phenomena that can be closer to or farther from the
> actual thing, we're talking about 'simulated information-processing'. Just
> looking at that phrase, it should be obvious that there is no distinction
> between 'simulated information-processing' and 'real
> information-processing', they are exactly the same thing.
>
> I'm starting to suspect that, clever as this is, there is no actual
> understanding of the issue (note that this doesn't contradict my original
> comment. Thinking and understanding are not the same thing). Or maybe it's
> a misunderstanding. But the overuse of superlatives is not only irritating,
> it's revealing. It's making me think that these things are still largely
> 'stochastic parrots'.
>
> I was going to say that we are anthropomorphising them, but it might be
> better to say that we are assuming that they're adults when they are
> actually very small children with a really large vocabulary, that have been
> listening to an enormous number of conversations. What they say sounds
> impressive, but they haven't yet developed the ability to actually
> understand things.
>
> --
> Ben
>
> _______________________________________________
> extropy-chat mailing list
> extropy-chat at lists.extropy.org
> http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20260909/c7039705/attachment.htm>