[ExI] chatbots and marketing: was RE: India and the periodic table

[Gadersd]

> Since I think you know more about the computing requirements to do the training than I do, I ask you: given a pool of a million volunteers with ordinary consumer-level CPUs but willing to allow the processes to run in the background, could not anyone with a pile of training material use that background computing resource and create a custom GPT?

The main challenge for this type of parallel computing is the way the architecture is designed for these models. Some computations, such as cryptocurrency mining, are easy to break into very small independent subprocesses. However, the transformer model underlying language models does not have this feature. A significant amount of communication must occur between the processes computing the transformer output. If the processes tightly coupled such as in a GPU this poses little issue. However, if the processes are distributed far apart in space the communication overhead destroys the rate of computation. This is a significant bottleneck for distributed background computing.

Most of the current transformer architectures are dense, meaning full connectivity. It may be possible to modify the architecture to make it more sparse, such as the switch transformer which Google has experimented with, to reduce the communication bottleneck.

> Next question for Gad or anyone else: is it not clear there is a huge market available?  I gave one example: a chat-bot trained on the books in the fish bowl, all of it in the 200 class in the old Dewey decimal system.  The result isn’t any good for any question outside what is found in the 200 class, but plenty of market exists and will persist, that is answered by the 200 class.

I think there is a market for personalized chatbots. The current proprietary chatbots are limited in the dialogs they are permitted to engage in. The only way for someone to experience the full extent of their fantasies is to interact with an unfiltered model. 

> On Jun 8, 2023, at 9:51 PM, spike jones via extropy-chat <extropy-chat at lists.extropy.org> wrote:
> 
>  
>  
> From: extropy-chat <extropy-chat-bounces at lists.extropy.org <mailto:extropy-chat-bounces at lists.extropy.org>> On Behalf Of Gadersd via extropy-chat
> Sent: Thursday, 8 June, 2023 6:29 PM
> To: ExI chat list <extropy-chat at lists.extropy.org <mailto:extropy-chat at lists.extropy.org>>
> Cc: Gadersd <gadersd at gmail.com <mailto:gadersd at gmail.com>>
> Subject: Re: [ExI] chatbots and marketing: was RE: India and the periodic table
>  
>> In any case… it is easy enough to see a GPT-toolkit coming, so that everyone can try training one’s own chatbot by choosing the training material carefully.
>  
> There is a huge difference between inference and training. You are correct that the masses can run some of the smaller models on CPUs and consumer GPUs. But that is just inference. Training these models requires much more compute. However, there have been quite a few quantization hacks that may enable training on low end hardware. I’m not sure how significant the tradeoff will be but the community has surprised me with the tricks it has come up with.
>  
> The wisdom used to be that one had to have a $12000 GPU to do anything interesting, but ever since the llama.cpp guy got a model running on a MacBook I think any declaration of hard limitations should be thrown out.
>  
> We may very well see an explosion of user trained models that blow up the internet…
>  
>  
>  
>  
> Hi Gadersd, 
>  
> Most of us have unused background processor power that is unused.  For many years I ran a math program called Prime95, where we were collectively searching for the next Mersenne prime.  There was SETI at home, which was analyzing signals from deep space looking for ET, but in all that, for about the past 30 years, I theorized that eventually the killer app would show up.  It would be something that takes mind-boggling amounts of CPU cycles to calculate, but something that is well-suited for doing in the background on many parallel processors.  Custom chatbots are an example (I think) of that magic process I have been anticipating for nearly three decades.
>  
> Since I think you know more about the computing requirements to do the training than I do, I ask you: given a pool of a million volunteers with ordinary consumer-level CPUs but willing to allow the processes to run in the background, could not anyone with a pile of training material use that background computing resource and create a custom GPT?  
>  
> Next question for Gad or anyone else: is it not clear there is a huge market available?  I gave one example: a chat-bot trained on the books in the fish bowl, all of it in the 200 class in the old Dewey decimal system.  The result isn’t any good for any question outside what is found in the 200 class, but plenty of market exists and will persist, that is answered by the 200 class.
>  
> Gad or anyone, could we  use volunteer background computing resources the way Prime95 has been doing all these years?
>  
> spike
>  
>  
> _______________________________________________
> extropy-chat mailing list
> extropy-chat at lists.extropy.org <mailto:extropy-chat at lists.extropy.org>
> http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat <http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20230608/2af85138/attachment.htm>