[ExI] Emily M. Bender — Language Models and Linguistics (video interview)

[Gordon Swobe]

Jason, I received a reply to my last to you, but before I dig into it, I
notice that it does not appear addressed to ExI. I am happy to reply
privately, but I think you probably meant it for the list.

In the meantime, I wanted to add to comments about emergent properties that
I have no argument with the idea that GPT might be exhibiting emergent
properties -- I agree it certainly appears that way -- but I would say they
are emergent properties of the grammatical relationships between and among
words, not evidence of any understanding of the meanings. I think Bender
actually touches on this subject in the interview but without actually
using the term "emergent properties."

It could very well be that something like this is one aspect of human
intelligence, but I think we also understand the meanings.

-gts





On Sun, Mar 26, 2023 at 2:56 AM Gordon Swobe <gordon.swobe at gmail.com> wrote:

> I have a smart home. Some of the iPhone apps associated with it have and
> can display what could be described as internal models representing my
> home. Does this mean these apps have a conscious understanding of the
> layout of my home? No, I think not, not as I and most people use the word
> understand. Only minds can understand things, and despite my home being
> "smart," I reject the idea that it has a mind of its own. That is nothing
> more than foolish science-fiction..
>
> -gts
>
> On Sun, Mar 26, 2023 at 12:01 AM Gordon Swobe <gordon.swobe at gmail.com>
> wrote:
>
>>
>>
>> On Sat, Mar 25, 2023 at 4:49 PM Jason Resch via extropy-chat <
>> extropy-chat at lists.extropy.org> wrote:
>>
>>> Hi Gordon,
>>>
>>> Thanks for sharing this video. I watched and and found the following
>>> points of interest:
>>>
>>> *1. She said they can't possibly be understanding as they are only
>>> seeing a sequence of characters and predicting distributions and what these
>>> models do is not the same thing as understanding language.*
>>> My Reply: These models demonstrate many emergent capabilities that were
>>> not things that were programmed in or planned. They can answer questions,
>>> summarize texts, translate languages, write programs, etc. All these
>>> abilities emerged purely from being trained on the single task of
>>> predicting text. Given this, can we be certain that "understanding" is not
>>> another one of the emergent capabilities manifested by the LLM?
>>>
>>
>> This gets into philosophical debate about what, exactly, are emergent
>> properties. As I understand the term, whatever it is that emerges is
>> somehow hidden but intrinsic prior to the emergence. For example, from the
>> rules of chess there emerge many abstract properties and strategies of
>> chess. To someone naive about chess, it is difficult to imagine from the
>> simple rules of chess how chess looks to a grandmaster, but those emergent
>> properties are inherent in and follow logically from the simple rules of
>> chess.
>>
>> So how does meaning emerge from mere symbols(words)? Sequences of
>> abstract characters in no possible way contain the seeds of their meanings,
>> as we can see by the fact that many different words exist in different
>> languages and in entirely different alphabets for the same meaning.
>>
>>
>>> *2. She uses the analogy that the LLM looking at characters would be the
>>> same as a human who doesn't understand Cherokee looking at Cherokee
>>> characters.*
>>> My Reply: This is reminiscent of Searle's Chinese Room. The error is
>>> looking at the behavior of the computer only at the lowest level, while
>>> ignoring the goings-on at the higher levels. She sweeps all possible
>>> behavior of a computer under the umbrella of "symbol manipulation", but
>>> anything computable can be framed under "symbol manipulation" if described
>>> on that level (including what atoms, or neurons in the human brain do).
>>> This therefore fails as an argument that no understanding exists in the
>>> higher-level description of the processing performed by the computer
>>> program.
>>>
>>
>> Yes her argument is similar to Searle's. See above. Sequences of
>> characters (words) in no possible way contain the hidden seeds of their
>> meanings, as we can see from the fact that many different words exist
>> in different languages and alphabets for the same meaning.
>>
>> *3. She was asked what a machine would have to do to convince her they
>>> have understanding. Her example was that if Siri or Alexa were asked to do
>>> something in the real world, like turn on the lights, and if it does that,
>>> then it has understanding (by virtue of having done something in the real
>>> world).*
>>> My Reply: Perhaps she does not see the analogy between turning on or off
>>> a light, and the ability of an LLM to output characters to a monitor as
>>> interacting in the real world (turning on and off many thousands of pixels
>>> on the user's monitor as they read the reply).
>>>
>>
>> I thought that was the most interesting part of her interview. She was
>> using the word "understanding" in a more generous way than I would prefer
>> to use it, even attributing "understanding" to a stupid app like Alexa, but
>> she does not think GPT has understanding. I think she means it in exactly
>> the way I do, which is why I put it in scare-quotes. As she put, it is a
>> "kind of" understanding. As I wrote to you I think yesterday, I will grant
>> that my pocket calculator "understands" how to do math, but it is
>> not holding the meaning of those calculations in mind consciously, which is
>> what I (and most everyone on earth) mean by understanding.
>>
>> Understanding involves the capacity to consciously hold something in
>> mind. Otherwise, pretty much everything understands something and the word
>> loses meaning. Does the automated windshield wiper mechanism in my car
>> understand how to clear the rain off my windows when it starts raining? No,
>> but I will grant that it "understands" it in scare-quotes.
>>
>> The other point I would make here is that even if we grant that turning
>> the pixels off and on your screen makes GPT sentient or conscious, the real
>> question is "how can it know the meanings of those pixel arrangements?"
>> From its point of view (so to speak) it is merely generating meaningless
>> strings of text for which it has never been taught the meanings except via
>> other meaningless strings of text.
>>
>> Bender made the point that language models have no grounding, which is
>> something I almost mentioned yesterday in another thread. The symbol
>> grounding problem in philosophy is about exactly this question. They are
>> not grounded in the world of conscious experience like you and me. Or, if
>> we think so, then that is to me something like a religious belief.
>>
>>
>>
>>> *4. She admits her octopus test is exactly like the Turing test. She
>>> claims the hyper-intelligent octopus would be able to send some
>>> pleasantries and temporarily fool the other person, but that it has no real
>>> understanding and this would be revealed if there were any attempt to
>>> communicate about any real ideas.*
>>> My Reply: I think she must be totally unaware of the capabilities of
>>> recent models like GPT-4 to come to a conclusion like this.
>>>
>>
>> Again, no grounding.
>>
>>
>>> *5. The interviewer pushes back and says he has learned a lot about
>>> math, despite not seeing or experiencing mathematical objects. And has
>>> graded a blind student's paper which appeared to show he was able to
>>> visualize objects in math, despite not being sighted. She says the octopus
>>> never learned language, we acquired a linguistic system, but the hyper
>>> intelligent octopus has not, and that all the octopus has learned is
>>> language distribution patterns.*
>>> My Reply: I think the crucial piece missing from her understanding of
>>> LLMs is that the only way for them to achieve the levels of accuracy in the
>>> text that they predict is by constructing internal mental models of
>>> reality. That is the only way they can answer hypotheticals concerning
>>> novel situations described to them, or for example, to play chess. The only
>>> way to play chess with a LLM is if it is internally constructing a model of
>>> the board and pieces. It cannot be explained in terms of mere patterns or
>>> distributions of language. Otherwise, the LLM would be as likely to guess
>>> any potential move rather than an optimal move, and one can readily
>>> guarantee a chess board position that has never before appeared in the
>>> history of the universe, we can know the LLM is not relying on memory.
>>>
>>
>> I don't dispute that LLMs construct internal models of reality, but I
>> cough when you include the word "mental," as if they have minds
>> with conscious awareness of their internal models.
>>
>> I agree that it is absolutely amazing what these LLMs can do and will do.
>> The question is, how could they possibly know it any more than my pocket
>> calculator knows the rules of mathematics or my watch knows the time?
>>
>>
>>
>>>
>>> *6. The Interviewer asks what prevents the octopus from learning
>>> language over time as a human would? She says it requires joint-attention:
>>> seeing some object paired with some word at the same time.*
>>> My Reply: Why can't joint attention manifest as the co-occurrence of
>>> words as they appear within a sentence, paragraph, or topic of discussion?
>>>
>>
>> Because those other words also have no meanings or refrents. There is no
>> grounding and there is no Rosetta Stone.
>>
>> Bender co-authored another paper about "stochastic parrots," which is how
>> she characterizes LLMs and which I like. These models are like parrots that
>> mimic human language and understanding. It is amazing how talented they
>> appear, but they are only parrots who have no idea what they are saying.
>>
>>
>>>
>>> *7. The interviewer asks do you think there is some algorithm that could
>>> possibly exist that could take a stream of words and understand them in
>>> that sense? She answers yes, but that would require programming in from the
>>> start the structure and meanings of the words and mapping them to a model
>>> of the world, or providing the model other sensors or imagery. The
>>> interviewer confirms: "You are arguing that just consuming language without
>>> all this extra stuff, that no algorithm could just from that, really
>>> understand language? She says that's right.*
>>> My Reply: We already know that these models build maps of things
>>> corresponding to reality in their head. See, for example, the paper I
>>> shared where the AI was given a description of how rooms were connected to
>>> each other, then the AI was able to visually draw the layout of the room
>>> from this textual description. If that is not an example of understanding,
>>> I don't know what possibly could be. Note also: this was an early model of
>>> GPT-4 before it had been trained on images, it was purely trained on text.
>>>
>>
>> This goes back to the question about Alexa.Yes, if that is what you mean
>> by "understanding" then I am forced to agree that even Alexa and Siri
>> "understand" language. But, again, I must put it in scare quotes. There is
>> nobody out there named Alexa who is actually aware of understanding
>> anything. She exists only in a manner of speaking.
>>
>>
>>>
>>> *8. She says, imagine that you are dropped into the middle of the Thai
>>> library of congress and you have any book you could possibly want but only
>>> in Thai. Could you learn Thai? The Interviewer says: I think so. She asks:
>>> What would you first do, where would you start? She adds if you just have
>>> form, that's not going to give you information. She then says she would
>>> have to find an encyclopedia or a translation of a book we know.*
>>> My Reply: We know there is information (objectively) in the Thai
>>> library, even if there were no illustrations or copies of books we had the
>>> translations to. We know the Thai library contains scruitable information
>>> because the text is compressible. If text is compressible it means there
>>> are discoverable patterns in the text which can be exploited to reduce the
>>> amount of bits needed to represent it. All our understanding can be viewed
>>> as forms of compression. For example, the physical laws that we have
>>> discovered "compress" the amount of information we need to store about the
>>> universe. Moreover, when compression works by constructing an internal toy
>>> model of reality, we can play with and permute the inputs to the model to
>>> see how it behaves under different situations. This provides a genuine
>>> understanding of the outer world from which our sensory inputs are based. I
>>> believe the LLM has successfully done this to predict text, it has various
>>> internal, situational models it can deploy to help it in predicting text.
>>> Having these models and knowing when and how to use them, I argue, is
>>> tantamount to understanding.
>>>
>>
>> How could you possibly know what those "discoverable patterns of text"
>> mean, given that they are in Thai and there is no Thai to English
>> dictionary in the Thai library?
>>
>> As she points out and I mentioned above, there is no Rosetta Stone.
>>
>> Thanks for the thoughtful email.
>>
>> -gts
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20230326/5bd390cc/attachment-0001.htm>