[ExI] ai in education

[John Clark]

On Fri, Mar 6, 2026 at 5:47 PM <spike at rainier66.com> wrote:

*> The AI’s version of safety might mean turning around and destroying the
> guy who fired the weapon. *
>

*You cannot be certain if that would be a good thing or a bad thing, but to
make the best decision you are capable of you would need to take into
consideration who ordered the guy to fire the weapon, and who designed the
safety features on the AI, and figure out which one was more trustworthy. *

*> The military needs to know exactly how an AI works,*
>

*If so then the military needs to resign themselves to the fact that they
will NEVER be able to use AI because even the people that made the AI don't
know exactly how it works, and in many cases not even approximately how it
works. And that's exactly why Eliezer is so worried. *


> *>which means the contracting company must turn over the source code. *
>

*That would be impossible because all AIs are neural networks and neural
networks  don't have a source code, they are a grid of interconnections in
which two or more inputs come together at a node and produce an output.
That output is determined by the inputs and by the "weight" that each node
has which is a 32-bit number and is constantly changing during the learning
process. A modern AI can have well over 1 trillion nodes. The AI companies
try to figure out the best basic way the wire things up but most of them
end up with some variation of the transformer idea.  The real magic comes
during the learning phase, they need to figure out the best way to expose
the neural network to the sum total of human knowledge. After you figure
all that out you put it into a hundred billion dollar computer and let it
run for about six months and then you've got a 32 bit number for every one
of those trillion+ nodes. *

*The AI company could give the military the complete wiring diagram and
tell them the 32 bit number of every one of the nodes, but if you pointed
to one node at random and asked why it has that particular number and not
some other number they couldn't begin to tell you. And if you asked what
the AI would do in a certain hypothetical situation then, if you're were
lucky, they might be able to say "I think it would probably do something
sort of like this" but in other hypotheticals, if they were being honest,
they would have to say "I have absolutely no idea, you'll just have to try
it and see". *

*  John K Clark*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20260307/5aa94b83/attachment.htm>