<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'> spike@rainier66.com <spike@rainier66.com> <br><b>Subject:</b> RE: [ExI] Maths Teachers Career Prospects<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'><o:p> </o:p></span></p><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'> <a href="mailto:spike@rainier66.com">spike@rainier66.com</a> <<a href="mailto:spike@rainier66.com">spike@rainier66.com</a>> <br><b>…<o:p></o:p></b></span></p></div><p class=MsoNormal><span style='font-family:"Arial",sans-serif'>…<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'>>…Many here have lived long. But did ANY of us foresee the day when a talented MATHEMATICIAN would be more popular than a rock star?<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'>spike<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'>I read many explanations for how LLMs work since about November 2022, but until one really groks the concept of cross entropy, one is missing an important link in the chain of understanding. Sanderson gives the best explanation of the concept I have seen:</span><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'><o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'><a href="https://www.youtube.com/watch?v=GlYgs6v2YfU">https://www.youtube.com/watch?v=GlYgs6v2YfU</a><o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'>Has anyone here seen a better explanation of that concept? Once you get cross entropy, does it not clarify why LLMs use so damn much power? And why I don’t think LLMs will ever run on cell phones? And why we need massive new power generation capacity to run data centers? And why we need billionaires to build the power plants and data centers? And why we need capitalism to build billionaires?<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'>spike</span><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'><o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif;color:black'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Arial",sans-serif;color:black'><o:p> </o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal> <o:p></o:p></p></div></body></html>