<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">You can try GPT-JT (not related to ChatGPT) <a href="https://huggingface.co/spaces/togethercomputer/GPT-JT" class="">https://huggingface.co/spaces/togethercomputer/GPT-JT</a>. Try it yourself and you will see that it is completely useless compared to ChatGPT. Note that this is a 6 billion parameter model trained using parallel computing as you have suggested. Even this small model is behind the capabilities of most consumer hardware: one needs a pricey GPU to run it. Running something as large as 175 billion parameters (ChatGPT) is impossible on consumer hardware.<div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div class="WordSection1" style="page: WordSection1;"><div class=""><div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">Sure, but my contention is that the distributed model would still multiply the same size matrix. If we need to multiply a 50x50, that task can be effectively distributed into background computing, but it would take reliable bandwidth and probably a lot of redundancy.</div></div></div></blockquote><div class=""><br class=""></div>The issue with this is that the transformer model uses quadratic matrix operations in addition to linear operations. The quadratic operations cannot be easily split across devices. It must be done on a single device (GPU) that has the memory to hold giant matrices. This is why one needs a legion of $10,000 GPU’s with massive memory to run ChatGPT level models.</div><div class=""><br class=""></div><div class="">It turns out that these quadratic operations are what enabled the intelligence of these models to far surpass previous techniques. The quadratic operations require the entire input memory to be collected in one place as this one step integrates every piece of information together. This reflects the nature of our intelligence: it is more than a sum of parts (linear).</div><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Mar 7, 2023, at 10:38 AM, spike jones via extropy-chat <<a href="mailto:extropy-chat@lists.extropy.org" class="">extropy-chat@lists.extropy.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div class=""><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0in 0in;" class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class="">From:</b><span class="Apple-converted-space"> </span>extropy-chat <<a href="mailto:extropy-chat-bounces@lists.extropy.org" class="">extropy-chat-bounces@lists.extropy.org</a>><span class="Apple-converted-space"> </span><b class="">On Behalf Of<span class="Apple-converted-space"> </span></b>Gadersd via extropy-chat<br class=""><br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [ExI] ChatGPT 'Not Interesting' for creative works<o:p class=""></o:p></div></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="" type="cite"><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">The year-old prediction is useless of course, but the idea is to compensate for the limited calculation ability and bandwidth by giving it more time.<o:p class=""></o:p></div></div></div></div></blockquote><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">>…The analogy does not extend to language models. You cannot compensate for a small model with more computing time. These models have a fixed computing burden that is inversely proportional to model size…<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">OK but the explanation you gave doesn’t support that contention. Read on please:<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">>…I think you have the wrong intuition … These models are essentially matrix multiplication. Small matrices multiply faster than large matrices…<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Sure, but my contention is that the distributed model would still multiply the same size matrix. If we need to multiply a 50x50, that task can be effectively distributed into background computing, but it would take reliable bandwidth and probably a lot of redundancy. <span class="Apple-converted-space"> </span><o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Consider the task of finding the determinant of a 50x50. That can be distributed among 50 computers each finding the determinant of a 49x49, each of which can be distributed into 49 processors and so on. Matrix multiplies and inversions can likewise be distributed, but of course it would be a brittle process: any one processor could mess it up.<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">OK idea: get a bunch of investors together who can kick in a few tens of thousands, rent some unused office or warehouse space somewhere, set up a closed system server farm training toward a particular bias agreed upon by the investors. You would form a scaled down (but still big) GPT which is intentionally trained in material friendly to libertarianism for instance, or believing that causing the extinction of mosquitoes is good but in general causing extinction is bad.<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Contention: whatever the ChatGPT investors did, a smaller group with less money can do likewise.<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Given that, one could create a commercial chatbot specialized in training students for instance, or spreading religion, or selling products. Oh I see mega profits trying to be made here.<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">spike<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div></div><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">extropy-chat mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><a href="mailto:extropy-chat@lists.extropy.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">extropy-chat@lists.extropy.org</a><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><a href="http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat</a><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""></div></blockquote></div><br class=""></div></body></html>