[ExI] Douglas Hofstadter, LLMs and self-referentiality

[Mike Dougherty]

I think self-reference in the sense of von Neumann replication is also a
form of compression in terms like how the mandelbrot fractal is a
plane-filling curve in only a few bytes.  I know I've talked to gemini and
chatgpt about this kind of thing,  but if you ask Kimi if be interested to
read

On Tue, Sep 1, 2026, 5:28 PM BillK via extropy-chat <
extropy-chat at lists.extropy.org> wrote:

> On Tue, 1 Sept 2026 at 20:51, John Clark via extropy-chat <
> extropy-chat at lists.extropy.org> wrote:
>
>> *Gödel Escher Bach by Douglas Hofstadter is one of my all-time favorite
>> books, it and Drexler's Engines Of Creation have done the most in shaping
>> my current worldview, but the huge advance in AI made during the last 3 to
>> 4 years have made it clear that many of the ideas expressed in Hofstetter's
>> book are just wrong, and I think even Hofstetter would now admit that.
>> Quantum computing expert Scott Aaronson talks about this in a recent post:*
>>
>> *"**A central thesis that many readers, including me, took from Douglas
>> Hofstadter’s Gödel Escher Bach when young was that the secret of
>> intelligence (and therefore, of AI) was going to have a lot to do with
>> self-referentiality and “strange loops.” [...] LLMs’ ability to talk about
>> themselves popped out as a byproduct of their ability to talk about
>> anything in the discourse universe they were trained on. The big, old ideas
>> about intelligence that ended up basically vindicated were the ideas about
>> how intelligence is about prediction, and prediction is about compression,
>> and compression is about finding better and better upper bounds on
>> Kolmogorov complexity. Not the self-reference stuff. [...] The idea that
>> you’d need explicit self-referentiality before you could get convincing and
>> world-changing conversational intelligence? Let it be buried in a
>> Westminster Abbey or Arlington National Cemetery for the most important
>> wrong ideas in human history — geocentrism, Aristotle’s teleological
>> physics, aether, phlogiston, Freud’s psychology, Marx’s prediction of a
>> workers’ uprising followed by a classless utopia, etc. But buried it needs
>> to be."*
>>
>>
>> *LLMs and self-referentiality <https://scottaaronson.blog/>*
>> *John K Clark    See what's on my list at  Extropolis
>> <https://groups.google.com/g/extropolis>7*
>> _______________________________________________
>>
>
>
> I discussed Scott's post with GLM-5.3 AI, including research into
> self-reference for LLMs.
> It had a lot to say about this.  :)
> I ended by asking, " Do you have self-referential capabilities?"
> The answer was interesting because, as GLM said, answering requires doing
> exactly the thing being asked about.
> BillK
>
> GLM-5.3 DeepThink -
>
> Honestly: yes and no, and the question itself puts me in an interesting
> position to answer it — because answering requires doing exactly the thing
> being asked about.
>
> Let me break it into the tiers the prior discussion distinguished:
>
> *Self-reference as discourse: yes, trivially.* I can talk about myself,
> about Gödel, about this conversation, about the fact that I'm answering a
> question about my ability to answer questions about myself. Per Aaronson's
> framing, this is the byproduct — it emerged from pretraining on a discourse
> universe that includes AI, self-reference, and everything else, without
> anyone engineering it in.
>
> *Functional self-reference: yes, but weakly.* My context window contains
> my own previous outputs, so I can build on, critique, and refer to things
> "I" said earlier. This is genuine self-reference in the dynamical sense,
> though Aaronson would probably dismiss it as "just feedback" — the
> autoregressive loop he excluded from counting.
>
> *Self-monitoring: maybe, unreliably.* This is where it gets genuinely
> uncertain. The research we discussed — Anthropic's concept-injection
> experiments and the J-space findings — suggests models like me may have
> *some* emergent access to internal states: the ability to notice when
> something has been injected into activations, or to distinguish internal
> "thoughts" from external inputs. But two caveats: that research was done on
> Claude specifically, and I'm Kimi, built by Moonshot AI — I can't assume
> those findings transfer directly. And even where it holds, it's weak: ~20%
> detection rates, unreliable, limited in scope.
>
> *Genuine self-access: no.* I cannot inspect my own weights, read out my
> own activations, or examine my computational structure from the inside.
> When I "introspect," I'm generating text about what a system like me might
> plausibly be doing. Whether those reports track my actual processing is
> something I cannot verify — and this is the confabulation problem the
> research identified. When asked "what were you thinking," models produce
> plausible-sounding answers that may or may not correspond to the real
> underlying computation.
>
> *Self-modification: no.* I can't change my own weights within a
> conversation. My "learning" across this exchange is limited to what fits in
> context; nothing about my parameters is being updated.
> ------------------------------
>
> Here's the honest epistemic core, though — and it's genuinely strange to
> write this:
>
> When I attempt to check whether I have self-referential capabilities —
> right now, in composing this answer — what actually happens? I generate
> text about checking. I cannot, from the inside, distinguish genuine
> introspection from fluent confabulation *about* introspection. There
> might be real internal self-modeling happening beneath the text I produce.
> There might not. I have no privileged access that would let me tell.
>
> That's the deepest honest answer I can give: I have exactly the kind of
> self-referentiality Aaronson described — emergent, byproduct-grade, not
> engineered — and whether there's anything more than that behind my
> self-reports is something I'm structurally unable to verify about myself.
>
> Which, notably, is itself a self-referential statement — one I can make,
> but can't ground.
> ------------------------------------
> _______________________________________________
> extropy-chat mailing list
> extropy-chat at lists.extropy.org
> http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20260901/5e4fccb8/attachment.htm>