[ExI] Douglas Hofstadter, LLMs and self-referentiality

[Jason Resch]

On Tue, Sep 1, 2026, 3:50 PM John Clark via extropy-chat <
extropy-chat at lists.extropy.org> wrote:

> *Gödel Escher Bach by Douglas Hofstadter is one of my all-time favorite
> books, it and Drexler's Engines Of Creation have done the most in shaping
> my current worldview, but the huge advance in AI made during the last 3 to
> 4 years have made it clear that many of the ideas expressed in Hofstetter's
> book are just wrong, and I think even Hofstetter would now admit that.
> Quantum computing expert Scott Aaronson talks about this in a recent post:*
>
> *"**A central thesis that many readers, including me, took from Douglas
> Hofstadter’s Gödel Escher Bach when young was that the secret of
> intelligence (and therefore, of AI) was going to have a lot to do with
> self-referentiality and “strange loops.” [...] LLMs’ ability to talk about
> themselves popped out as a byproduct of their ability to talk about
> anything in the discourse universe they were trained on. The big, old ideas
> about intelligence that ended up basically vindicated were the ideas about
> how intelligence is about prediction, and prediction is about compression,
> and compression is about finding better and better upper bounds on
> Kolmogorov complexity. Not the self-reference stuff. [...] The idea that
> you’d need explicit self-referentiality before you could get convincing and
> world-changing conversational intelligence? Let it be buried in a
> Westminster Abbey or Arlington National Cemetery for the most important
> wrong ideas in human history — geocentrism, Aristotle’s teleological
> physics, aether, phlogiston, Freud’s psychology, Marx’s prediction of a
> workers’ uprising followed by a classless utopia, etc. But buried it needs
> to be."*
>


I never viewed Hofstadter's strange loops as being necessary for nor
central to intelligence. Rather in my readings of G.E.B. I see
self-reference and strange loops as the origin of mysteries of the "I" and
central to the understanding of how things like free will emerge when the
direction of causality gets twisted in strange ways (generally up from low
levels influencing higher levels) but then becoming "strange" when high
levels in that hierarchy reach back down to influence the low levels again,
creating what Hofstadter called a "tangled hierarchy".

Tangled hierarchies are important in both the brain (I can tell you to
blink, and your high level thoughts reach back down to influence low level
motor neurons) and likewise for a LLM to think or generate more than a
token of text, it too just loop back around to send its highest level
output back down to the low levels.

Here are some quotes I think are relevant from Hofstadter and others on
this:

"Our thoughts seem to run about in their own space, creating new thoughts
and modifying old ones, and we never notice any neurons helping us out! But
that is to be expected. We can’t. […]
We should remember that physical law is what
makes it all happen–way, way down in neural nooks and crannies which are
too remote for us to reach with our high-level introspective probes."
— Douglas Hofstadter in “Gödel, Escher, Bach” (1979)

This points out that we are as ignorant of what our own neurons are doing
as LLMs are ignorant of what their own low level activations and weights
are doing. Minds, being high level phenomena, will generally lack access
and introspection to the low levels supporting them.

"Gödel’s Theorem follows from a consequence of the general fact: I am
trapped inside myself and therefore can’t see how other systems see me.
Thus the objectivity-subjectivity dilemmas that Nagel has sharply posed are
somehow related to epistemological problems in both mathematical logic, and
as we saw earlier, the foundations of physics."
— Douglas Hofstadter and Daniel Dennett in “The Mind’s I” (1981)

I think this insight is powerful. It extends the known mathematical limits
of self reference to explain the incommunicability and unknowability of the
qualia of other systems. In other words, self reference (and it's
limitations) answer Nagel's question of why he cannot know what it is like
to be a bat, unless he is a bat, but then he is no longer Nagel. So Nagel
(the system) is trapped in itself, and thus cannot access bat-like
knowledge.

"In order to deal with the full richness of the
brain/mind system, we will have to be able to slip between levels
comfortably. Moreover, we will have to admit various types of “causality”:
ways in which an event at one level of description can “cause” events at
other levels to happen. Sometimes event A will be said to “cause” event B
simply for the reason that the one is a translation, on another level of
description, of the other. Sometimes “cause” will have its usual meaning:
physical causality. Both types of causality–and perhaps some more–will have
to be admitted in any explanation of mind, for we will have to admit causes
that propagate both
upwards and downloads in the Tangled Hierarchy of mentality […]."
— Douglas Hofstadter in “Gödel, Escher, Bach” (1979)

I regard this as generally true. When we speak of a person wanting a glass
of milk, and going to get it, this is still meaningful, even though
everything could fundamentally be explained in terms of interacting quantum
fields, and no laws of physics are violated by a person's mind or will,
without this insight, we face a dilemma in explaining what the point of
consciousness is if it "doesn't do anything" above and beyond what the mere
atomic forces and interactions alone accomplish. Here we see the mind as
causally efficacious, just as we see molecules, cells, and neural networks
as each operating within their own level and domain of causality.

Thoughts can affect thoughts, just as the thoughts on my brain are now
reaching and affecting thoughts in your brain.

"My belief is that the explanations of “emergent” phenomena in our
brains–for instance, ideas hopes, images, analogies, and finally
consciousness and free will–are based on a kind of Strange Loop, an
interaction between levels in which the top level
reaches back down towards the bottom level and influences it, while at the
same time being itself determined by the bottom level."
— Douglas Hofstadter in “Gödel, Escher, Bach” (1979)

Note Hofstadter here does not invoke strange loops to explain intelligence.
Instead his focus is on its lower to explain and better understand
consciousness and free will.

"Perhaps consciousness arises when the brain’s simulation of the world
becomes so complete that it must include a model of itself. Obviously the
limbs and body of a survival machine must constitute an important part of
its simulated world; presumably for the same kind of reason, the simulation
itself could be regarded as part of the world to be simulated. Another word
for this might indeed be “self-awareness,” […]
— Douglas Hofstadter and Daniel Dennett in “The Mind’s I” (1981)

This but sounds almost exactly like what Aaronson said. Once the model of
the world is complete enough, it must necessarily include some kind of
model of itself, as an agent acting and existing within the world, itself
being something it must also explain and predict, or at least contain a
sort of self referential notion of. Note the insurances where the LLM uses
the word "me" within its answers. That is an example of this sort of
modeling self-reference.


Some other (possibly useful) insights from others on self reference:

"When people talk about consciousness, something often mentioned is
“self-awareness” or the ability to “think about one’s own processes of
thinking”. Without the conceptual framework of computation,
this might seem quite mysterious. But the idea of universal computation
instead makes it seem almost inevitable. The whole point of a universal
computer is that it can be made to emulate any computational system—even
itself."
— Stephen Wolfram in “What is Consciousness” (2021)

"The power of recursion evidently lies in the
possibility of defining an infinite set of objects by a finite statement.
In the same manner, an infinite number of computations can be described by
a finite recursive program, even if this program contains no explicit
repetitions."
— Niklaus Wirth in “Algorithms + Data Structures = Programs” (1976)

"There was a young man who said, “Though,
it seems that I know that I know,
what I would like to see
is the ‘I’ that knows ‘me’
when I know that I know that I know.”"
— Alan Wattsin “The Book On the Taboo Against Knowing Who You Are” (1966)

"The soul is that to which one refers when one says ‘I’."
— Ibn al-Nafīsin “Theologus Autodidactus of Ibin Al Nafis” (1288)

"To exist, [consciousness] must have cause–effect power; to exist from its
own intrinsic perspective, independent of extrinsic factors, it must have
cause–effect power upon itself: its present mechanisms and state must ‘make
a difference’ to the probability of some past and future state of the
system (its cause–effect space), independent of extrinsic factors."
— Giulio Tononi and Christof Koch in “Consciousness: here, there and
everywhere?” (2015)

"It is not because some “information processing” could support
consciousness that we can conclude that all information processing can
support consciousness. You need at least one reflexive loop.
You need two reflexive loop for having self-
consciousness (Löbianity)."
— Bruno Marchal in discussion list (2019)

"In the end, we self-perceiving, self-inventing,
locked-in mirages are little miracles of self-
reference."
— Douglas Hofstadter in “I Am a Strange Loop” (2007)


As a process, it’s not always easy to tell where a mind-process ends and
where its exterior environment begins.

Consider that: over time, a thermostat can be affected by its own previous
actions. Does this cyclical action over time qualify as having cause–effect
power over itself?

Likewise, in large language models (or LLMs), a single invocation is purely
a feed-forward neural network — devoid of loops. But to generate more than
a few characters of text, the output of this network must be fed back in as
input to the next cycle of invocation. This thereby forms a reflexive loop.

If we could ask the LLM where its output text came from, the only valid
answer it could give would have to involve some kind of reference to itself.

As Dennett speculated, such reflection may be the path to consciousness:

"Is the process of unconscious reflection, then, a path by which a zombie
could turn itself into
[something that monitors its own activities,
including even its own internal activities, in an
indefinite upward spiral of reflexivity], and thereby render itself
conscious?"
— Daniel Dennett in “Consciousness Explained” (1991)

Computational process that keep a memory updated across time, hence, any
Turing machine, can be viewed equivalently as recursive functions.

Recursion and self-reference are then, indispensable features of almost
every computation.

Jason


>
>
> *LLMs and self-referentiality <https://scottaaronson.blog/>*
>
> *John K Clark    See what's on my list at  Extropolis
> <https://groups.google.com/g/extropolis>*
>
> *7ui*
>
>
>
>
> _______________________________________________
> extropy-chat mailing list
> extropy-chat at lists.extropy.org
> http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20260902/a1556d02/attachment.htm>