[ExI] Language Models on Consumer Hardware

[BillK]

On Sun, 12 Mar 2023 at 22:06, Gadersd via extropy-chat
<extropy-chat at lists.extropy.org> wrote:
>
> Someone managed to run the 13 billion parameter model on a 3.6 GHz processor with 8 GB ram so those phone specs are close! But be aware that GPT3 is 165 billion parameters so we are still a while from running something as big as ChatGPT on a phone. The 13 billion parameter models are approaching usability though.
> _______________________________________________


You can now run a GPT-3 level AI model on your laptop, phone, and Raspberry Pi
Thanks to Meta LLaMA, AI text models have their "Stable Diffusion moment."
Benj Edwards - 3/13/2023

<https://arstechnica.com/information-technology/2023/03/you-can-now-run-a-gpt-3-level-ai-model-on-your-laptop-phone-and-raspberry-pi/>
Quotes:
Things are moving at lightning speed in AI Land. On Friday, a software
developer named Georgi Gerganov created a tool called "llama.cpp" that
can run Meta's new GPT-3-class AI large language model, LLaMA, locally
on a Mac laptop. Soon thereafter, people worked out how to run LLaMA
on Windows as well. Then someone showed it running on a Pixel 6 phone,
and next came a Raspberry Pi (albeit running very slowly).
If this keeps up, we may be looking at a pocket-sized ChatGPT
competitor before we know it.

Typically, running GPT-3 requires several datacenter-class A100 GPUs
(also, the weights for GPT-3 are not public), but LLaMA made waves
because it could run on a single beefy consumer GPU. And now, with
optimizations that reduce the model size using a technique called
quantization, LLaMA can run on an M1 Mac or a lesser Nvidia consumer
GPU.

Right now, our only guarantee is that things will change rapidly.
----------------------

BillK