[ExI] GATO

[Stuart LaForge]

Quoting Rafal Smigrodzki <rafal.smigrodzki at gmail.com>:

> I commented on the Lex Friedman interview with Oriol Vinyals on YouTube:
>
> The tokens seem to be analogous to the patterns of neural activation in the
> brain that are sent over long-range intracortical pathways. Long-range
> connections underlie conscious processing and allow cross-modality
> inference. Packets of information sent over the long-range connections
> represent usually the highest-level features recognized by local cortical
> networks and recurrent processing of such features in multiple cortical
> areas allows the extraction of ever more complex and abstract
> representations of multi-modality patterns of information. This is
> fascinatingly similar to GATO which extracts features of inputs in
> different special purpose modules in the form of tokens and proceeds to
> extract higher level features by juxtaposing tokens sent from different
> modules. Very clever and very brain-like.
>
> --
> Rafal Smigrodzki, MD-PhD
> Schuyler Biotech PLLC

It is of note that GATO is a transformer neural network very similar  
to the natural language models like GPT-3 and LAMDa, only it was  
trained with other forms of data in addition to text. It sounds like  
you are talking about attention tokens which are a form of short-term  
memory for transformer neural networks. In the language models, it  
basically allows the the AI to remember and reference the contextual  
topic so that its writing seems coherent from beginning to end. In  
GATO they probably facilitate the "cross-modality inference". The  
middle layers of a neural network module are very abstract and if they  
were sharing some attention tokens, they could very easily juxtapose  
multiple sensory modalities into some kind of abstract motif or  
gestalt as it were.

Interesting observation, Rafal. :)

Stuart LaForge


Stuart LaForge