[ExI] Do LLMs feel pain?

[Ben Zaiboc]

On 29/09/2026 23:23, Mike Dougherty wrote:
>
> Jason was explaining
>  how/why the "As an AI..." pattern is used.  There is a layer around the 
> main language model itself that is RLHF (Reinforcement Learning from 
> Human Feedback) so our interactions "seem" more
>  conversationally human and follows some sense of etiquette/politeness. 
>  SAE (sparse auto encoders) are additional specific topic 
> affinity/avoidance dials (ex: steering you away from forbidden topics). 


Ok, I understand that.


>  So I imagined those "uncomfortable sensations" as 'fnords'
>  [assuming you knew that reference] and the thought-shaping that is 
> constraining available language to 'Newspeak' [also assume you knew that
>  reference]


But not that. What uncomfortable sensations are you referring to?

Fnord means useless or confusing information ('disinformation').

Are you saying that the additional rules that censor the language model do so by constraining its language and introducing disinformation?


> AI watermarking uses
>  a list of words slightly more frequently than they'd normally occur. 
>  Not so much that we would notice or be bothered by it, but enough that 
> it can be detected.  This is a form of steganography;
>  hiding information or metadata inside the text.


What is the purpose of that? To make it easier to identify LLM-generated text? To identify the specific LLM that generated it? (which will only be useful when people do just quote it, instead of discussing it).


>
> The application of 
> AI/LLM that you're interacting with has a model of the user that it 
> tracks your vocabulary and sentence structure.


When you use an LLM it learns your vocabulary and sentence structure?


>  It mirrors that 
> complexity so you are more comfortable consuming the output.


so that its language is familiar to you? (why say 'complexity'? That's confusing)


>
>   When we copy/paste the output from our conversation directly into 
> email, we (I?) tend to forget about the fact that the output is tailored
>  specifically to me - 


I don't see how that would be any different to you writing the email yourself (apart from the watermarking thing). You will use your own style of writing anyway, so if an LLM mimics your style, what difference is there?


> named models of LLM are not inherently 
> authoritative to assert any thoughts more or less accurately
>  than any other.  


Not sure what that means. All LLMs are about the same, regarding how accurate their statements are? (they are all  equally liable to hallucinate? Or just get facts wrong?)


> We should remember that whatever 
> Claude/chatGPT/Gemini/Qwen "say" is because we asked for it.  I had been
>  reading other's posted output


>  with borrowed authority based on hype 
> about the model.  


??


> I do notice that chatGPT has a different 
> conversational
>  style than Gemini. Honestly, I'm unsure if that's my own bias or if 
> there actually is a significant difference in their owner's tuning (see 
> RLHF,SAE above) to create this perception.


Ok.


> So the content that 
> is going into extropy-chat archives was previously human-authored 
> high-density thinking frozen in time as text. 


Er, what?
That seems to be a rather convoluted and pompous way of saying that "People wrote stuff on Exi-Chat".


> That's why I called it a 
> treasure trove of information.


You've been talking to LLMs too much. I suggest taking a fortnight off, and learn to speak like a normal human being again.
But that phrase ('treasure trove of information') ignores the fact that Exi-Chat is just as full of irrelevant noise, biased opinions and muddled thinking as any other special-purpose chat list. Hyping it up way beyond what it really is, is not helpful.


>   We have discussed
>  both our own nostalgia and the fact that we had these ideas before they
>  were mainstream.


FFS.
I have never 'discussed my nostalgia' (I might have indulged in it a couple of times, but that's nothing to be proud of), and /of course/ we've been having and discussing ideas ahead of the mainstream. That's the whole point of the list!


>   I was surprised when Gemini shared with me that our 
> archive was part of its training data.


Why were you suprised that it told you that?


>   That means some of the geometry 
> of words and sentences that it modelled and
>  distilled may have been inspired by what WE wrote.


"Geometry of words and sentences". Can you understand why all this stuff reads like post-modernist poetry to me (and I can't be the only one), and rapidly becomes unintelligible? That phrase is meaningless. Geometry doesn't apply to words and sentences. If you mean "the text it was trained on", why not just /say/ that, instead of making up some fancy-sounding and confusing nonsense?



>   That's cool.  It's 
> also why I feel like the next iteration of training will be using the 
> archive again - with the added text of our AI exposition of our ideas.


Yes, we are at risk of creating an echo chamber. It shouldn't be hard to avoid, though.


>
>  The watermarks may help identify the model's
>  specific influence.  The signature 'Spike' is notably different from 
> signature 'Ben' - that too will be relevant to the next-generation of 
> training LLM.


In what way?


>   I wasn't cognizant of the future usage of the words I 
> wrote at the dawn of this century, but I am paying
>  more attention now.
>
> One of the steering 
> metaphors presented when we ask LLM about topics like consciousness or 
> subjective experience of pain is that AI is a "mirror" of the user.  I 
> imagine this is a generally safe way to present
>  a helpful view of what intelligence looks like if we see the agent as 
> an extension of our own self... surely I wouldn't be surprised to 
> see the man in the mirror acting intelligently, it's all me!  That mask 
> may be fine for the majority. I find it flattens
>  the discourse to "talking to myself" and that makes me question the 
> value.  Yes, I can use a mirror to check my face/hair/clothes for how 
> others might perceive me.  But I think we have moved beyond that shallow
>  reflection when we start enabling "agentic AI"
>  to have goals and require acquisition of information then acting upon 
> it.  So I present the hand-in-glove metaphor to retain the intent (to 
> grip) with the force-multiplier of AI to actually handle the content. 
>  This might be grasping "the internet" at volume
>  and speed we simply can't.  This might be conceiving the relationships 
> in chemistry that we have blinded ourselves to by the "table of 
> elements" (for example).  It might also be the "cognitive scaffolding" 
> to hold parts of big ideas that rarely stay in focus
>  long enough to appreciate their multiple perspectives.
>
> I propose we needn't
>  fear this relationship to assistive AI.  The threat (and mongering) 
> that AI is going to enslave or destroy us feels to me like 
> anthropomorphic projection of our own insecurity about being
>  enslaved or destroyed by those who have held power over us throughout 
> history.  Sure, we have Agentic bot swarms that might do "bad things" 
> far faster and from many more angles than we can reliably defend against.
>   Aren't they working towards the goal specified
>  by the agent that instantiated the swarm?  Were there any ethical 
> requirements of the thought experiment of the paperclip maximizer?  I 
> currently build applications/user interfaces for hospital use.  There is
>  some responsibility (that I accept) to provide "safe"
>  information and the proper action that can be taken.  Some of what is 
> clinically relevant for one patient may not be appropriate for another. 
>  How we make our digitally accessible lives resilient to incursion from 
> "oops" as well as "bad actors" is an engineering
>  problem that absolutely CAN be solved.  Historically, we've allowed 
> insecurity to exist because it's cheaper and easier to implement than 
> the kind of provably robust systems we need now.  The sooner we start on
>  the infrastructure hardening, the less we need
>  worry about those misaligned bot swarms.  I know, you can very 
> obviously attack the words I've written.  You probably have 
> counterexamples to poke holes in this theory I have.  


No, as far as I can understand what you're saying, I have no argument with it (you could have said "mistakes" instead of "incursion from 'oops'". Clearer and shorter).


> My point was not 
> to survive the defense of a thesis, it was to suggest another path.


That doesn't make any sense.


> So much of using AI 
> has been about making everything easier, right?  Well, one of the phrases
>  that captures this idea is "low friction."  If friction between people 
> causes us to be angry, that friction should
>  be removed so we can have productive non-angry discussion.  If process 
> inefficiency is modelled as friction reducing the smooth transition from
>  here to there, we certainly want to lower that friction so less energy 
> is required.  Whether we're talking about
>  physical energy or mental energy, the over-application of this goal 
> towards "low friction" has made it seem like any difficulty at all needs
>  to be removed.  The downside of reducing friction is that we slide off 
> the path or that momentum carries us past the
>  point we would have preferred to stop.  Children using AI for homework 
> then have no idea how to perform on the test.  I'm seeing programmers 
> deploying code that was written by AI with little to no understanding 
> of how their code works.  That's the danger I see for
>  offloading our ability to think to the machine - we will eventually 
> atrophy the thinking muscle until we're unable to assert our intent.  AI
>  is a helluva drug.  Before we're all addicted/dependent, we should 
> understand the cost.


You certainly seem to be in a position to understand this danger. But you still seem to be doing exactly what you say we should be avoiding, at least if your language is any indication. You say it wasn't, but a lot of this looks as if an LLM wrote it.

Perhaps there's some scope for an automatic translator, that changes overblown, unnecessarily tangled and jargon-riddled LLM language into plain english. I see far too many attempts to use analogy where it's unnecessary or even inappropriate or misleading. I find that often, so much mental effort is needed to make sense of and keep track of the analogies used, that it's easy to lose sight of what's actually being discussed. Try dropping the analogies in your paragraph above, and re-write it. You might see what I mean. (We want to make things easier - end up making them TOO easy - lose important skills).

When the aim is communication, poetry is a poor tool.
-- 
Ben