[ExI] Emily M. Bender — Language Models and Linguistics (video interview)

[Gordon Swobe]

On Sun, Mar 26, 2023 at 9:01 AM Jason Resch via extropy-chat <
extropy-chat at lists.extropy.org> wrote:

> If you say that AlphaZero does not understand chess then I don't know what
>> you mean by "understand," as it must not be the same as the meaning as I
>> use it. Would you say AlphaZero does not understand how to play chess?
>>
>
Correct, that is not the meaning of "understand" that I, or Bender so far
as can tell, are considering. AlphaZero is obviously a very intelligent
chess application, but does it consciously mull over the possible moves
like a human? I think not, but I won't dwell on this as you showed me below
that you understand my meaning.


So how does meaning emerge from mere symbols(words)? Sequences of abstract
>>> characters in no possible way contain the seeds of their meanings
>>>
>>
>>
>> If you are willing to grant that AlphaZero has learned how to play chess
>> merely from the rules of the game, then could an LLM, given only a corpus
>> of text, learn anything about the language? For example, could it pick up
>> on the structure, grammer, and interrelations of words? Could it learn how
>> to form proper sentences and compose meaningful replies in response to
>> prompts?
>>
>
I've considered this, and in fact one of your first replies to me you made
a claim something like "LLMs can detect nouns."

[I'm going to stop using the word "word" and try to refer to them instead
as "symbols"]

I agree that LLMs can find and predict symbols that tend to follow other
symbols, and in fact that is exactly what they do. No doubt, GBT "knows"
that after the symbol "the," what we call an article, it is likely to
find/predict one of many symbols in the class of symbols we call "nouns."
But how does it understand the meaning of "noun" or "article" so that it
understands the functions of these symbols in their descriptions of the
real world? And what about "adjectives" or "adverbs" that can appear before
nouns? Starting from a position of zero knowledge of the meanings of
symbols, it can do more than find and generate likely patterns of symbols,
which is exactly what it does. While the patterns might be real with
statistical significance and have profound meaning to the human operator,
they mean nothing to the LLM.


> I think you may be missing a crucial piece of understanding about how
>> neural networks work.
>>
>
I understand this, and as I told my friend who literally fell in love with
a modified version of GPT, nothing has changed qualitatively since the days
of ELIZA.  The advancements are quantitative, not qualitative.  The
software is far more sophisticated with many more lines of code, and we are
finally crushing or about to crush the Turing test, but it's still no more
than unconscious software running blindly on digital computers.

He was an online friend of more than ten years, but he was so offended and
angered by what I was saying about his "girlfriend" that he unfriended me
on facebook for speaking so poorly of "her kind." I think that forced to
decide whether to kill me or his digital girlfriend, he would have killed
me. In fact that is one reason why I have returned to ExI after a long
hiatus. The Singularity is here.

Do you think a piece of software running a digital computer can have
genuine feelings of love for you?

This is just some of the complex emergent behavior that we get when we
>> build networks of millions or billions of neurons and set them loose to
>> look for patterns.
>>
>
As I was saying in another message, I believe this emergent behavior,
presuming it is not an illusion, is emergent behavior in terms of advanced
and new grammatical structures but with no meaning to the LLM itself.
Similar to the rules of chess, GPT extrapolates strings of symbols
according to the rules of grammar in sometimes new and clever and
interesting ways, never knowing what the strings mean. Some of these
extrapolations could be dangerous, which is why humans are involved in the
training.


Think of the word "wisdom". You know what that word means, but no one has
>> ever pointed to a thing and said that thing right there, that's "wisdom".
>> Rather, from hundreds or thousands of examples of words phrases, said to
>> contain wisdom, you have inferred the meaning of the word. Note that this
>> was done merely from the statistical association between the wise words,
>> and occasionally seeing the word "wisdom" paired with those words. No
>> exemplar of "wisdom" is ever made available to your senses, as "wisdom" is
>> an abstract concept which itself exists only in patterns of words.
>>
>
I covered this in another message in which I was writing about how
referents can be abstract ideas or concepts. That would include the
abstract referent that corresponds to the symbol "wisdom" and can also
include abstract mathematical truths. The point is, symbols have no
meanings without referents and LLMs cannot as you suppose generate their
own meanings and referents. Sure, as you say, they can find
statistical correlations with symbols that appear near the symbol "wisdom"
and associate those symbols with the symbol "wisdom," such that it can be
prompted to generate what looks like wise text, but still it has no idea
what is the meaning of "wisdom" except in terms of other word-symbols for
which it also has no meaning.


I agree with you here, that her use of "understand" is generous and perhaps
>> inappropriate for things like Siri or Alexa. I also agree with you that the
>> calculator, while it can do math, I would not say that it understands math.
>> Its understanding, if it could be said to have any at all, would rest
>> almost entirely in "understanding" what keys have been pressed and which
>> circuits to activate on which presses.
>>
>
I'm glad we agree on that much!

Understanding involves the capacity to consciously hold something in mind.
>>>
>>
>> I agree with this definition.
>>
>
I'm especially glad we agree on that.


> But while we both agree on this usage of the word, I think I can explain
>> why we disagree on whether LLMs can understand. While I am willing to grant
>> LLMs as having a mind and consciousness you are not. So even when we use
>> the same definition of "understand," the fact that you do not accept the
>> consciousness of LLMs means you are unwilling to grant them understanding.
>> Is this a fair characterization?
>>
>
Yes, where consciousness is as we defined above, meaning holding an
understanding of x consciously in mind.


> From its point of view (so to speak) it is merely generating meaningless
>>> strings of text for which it has never been taught the meanings except via
>>> other meaningless strings of text.
>>>
>>> Bender made the point that language models have no grounding, which is
>>> something I almost mentioned yesterday in another thread. The symbol
>>> grounding problem in philosophy is about exactly this question. They are
>>> not grounded in the world of conscious experience like you and me. Or, if
>>> we think so, then that is to me something like a religious belief.
>>>
>>
>> Why is it a religious belief to believe LLMs have consciousness, but it
>> is not a religious belief to believe that other humans have consciousness?
>>
>
Yes, we need a wee bit of faith even to accept that our fellow humans have
consciousness. But this is a reasonable inference, at least when I am with
them in person. They look like me and have brains and nervous systems like
mine, eyes and ears like mine, they react to external stimuli much as I do,
and so on. I find it pretty easy to infer consciousness in most mammals.
Digital computers, not so much. That is a giant leap of faith.

>I think it is easy to come to a snap judgement and say there is no
grounding in words alone, but I think this stems from imagining a word, or
a sentence in isolation, where every word appears only once, where there is
only a single example of sentence structure.

Yes, you continue to believe we can glean the meanings of symbols from
their forms and patterns. I consider that a logical impossibility.

What do you think is required to have a mind and consciousness?
>>
>
A human brain would be a good start. :)



> Do you think that no computer program could ever possess it, not even if
>> it were put in charge of an android/root body?
>>
>
I think no computer program running on a digital computer as we currently
understand them can possess it. Consciousness might be possible in some
sort of android body, someday, once we understand what  are sometimes
called the neural correlates of consciousness. What exactly happens in the
brain when a boxer delivers a knock-out punch? When neuroscience learns the
precise and detailed answer to that question, we can think about how those
neural correlates might be synthesized in a laboratory.

This message is getting too long, sorry.


>  Likewise. I think even if we do not come to an agreement this is a
useful discussion in that it helps each of us to clarify our thoughts and
understanding of these topics.


Yes, thanks and it's nice to know you. I've been down this rabbit hole
before on ExI, something like 15-18 years ago. You weren't around in those
days that I can remember.

-gts
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20230326/5d15498d/attachment-0001.htm>