[ExI] Why do the language model and the vision model align?

[John Clark]

On Wed, Feb 11, 2026 at 10:29 AM Jason Resch via extropy-chat <
extropy-chat at lists.extropy.org> wrote:

* > consider that the brain only gets neural spikes, from different nerves
> at different times.These are just symbols, bits.*
>

*Yes, and a neuron cell does not understand how the external physical world
works; the entire individual does but he can directly interact with that
world.   *


> *> And yet, from these mere patterns of firings, the brain is able to
> construct your entire world.This test is no less magical than what LLMs can
> do from "just text".*
>

*I don't think either one is doing anything "magical", it's just that I
don't have a deep understanding of how they are able to do what they do,
and most of the top researchers at the AI companies admit that they don't
either.*


*>> OK, I can see how you might be able to determine that a book written by
>> ET was about arithmetic, and that some of the symbols represented integers
>> and not letters, and that a base 10 system was used. But I don't see how
>> you could use the same procedure to determine that another book was about
>> chemistry unless you already had a deep understanding of how real world
>> chemistry worked, and I don't see how you could obtain such chemical
>> knowledge without experimentation and by just looking at sequences of
>> squiggles. But apparently there is a way. *
>>
>
> *> A student can learn a great deal from reading a chemistry text books,
> without ever entering a lab and taking out a beaker.*
>

*Yes but before the average student has even open a chemistry book he has
already been exposed to nearly two decades of real world experience and has
an intuitive feel for things like mass, velocity, heat and position, so he
can understand what the book is saying, but that would not be the case for
an AI that has never observed anything except a sequence of squiggles and
understood almost nothing except arithmetic and the laws of logic. It might
have an intuitive understanding of time but not of space, and from such a
humble beginning I don't understand how anything could deduce Newtonian
Physics, much less Quantum Physics which would be required to understand
modern chemistry. *


*>> The existence of isotopes would greatly complicate things, for example
>> we know that the element Tin has 10 stable isotopes and 32 radioactive
>> ones, they all have identical chemical properties but, because of neutrons,
>> they all have different masses. *
>>
>
> *> The periodic table, and Wikipedia's article on each element, lists
> atomic number (number of protons) in addition to atomic weights.*
>

*But how could the poor AI make sense out of that Wikipedia article if it
had no understanding of what the sequence of squiggles "w-e-i-g-h-t-s" even
means?  I don't deny that it can understand what it means, I just don't
know how.  *

>
*>> Perhaps Mr. Jupiter Brain could deduce the existence of the physical
>> world starting from nothing but arithmetic (but I doubt it) however it is
>> certainly far far beyond the capability of any existing AI, so they must be
>> using some other method. I just wish I knew what it was. *
>>
>
>
> *> Are you familiar with the universal approximation theorem?*
>

*Yes, a neural network can model any continuous function with arbitrary
precision, but the vast majority of continuous functions do not model
anything fundamental in either Newtonian or Quantum Physics, so** how does
an AI differentiate between those that do and those that don't?  *

> "What does it mean to predict the next token well enough? [...] It's a
> deeper question than it seems. Predicting the next token well means that
> you understand the underlying reality that led to the creation of that
> token."
> -- Ilya_Sutskever (of OpenAI)
> https://www.youtube.com/watch?v=YEUclZdj_Sc
>

*Ilya Sutskever certainly knows a hell of a lot more about this than I do
so maybe he's right.*



* John K Clark *
>
>
>
>
>>
>
> * By chance you might be using the best possible compression algorithm on
>> your data, but there's no way to prove to yourself or to anybody else that
>> you are.*
>>
>> *  John K Clark*
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20260212/2ad35e7e/attachment.htm>