[ExI] ai in education

[Adrian Tymes]

On Sat, Mar 7, 2026 at 10:47 AM spike jones via extropy-chat
<extropy-chat at lists.extropy.org> wrote:
> From: John Clark <johnkclark at gmail.com>
> Subject: Re: [ExI] ai in education
>
> On Fri, Mar 6, 2026 at 5:47 PM <spike at rainier66.com> wrote:
> > The AI’s version of safety might mean turning around and destroying the guy who fired the weapon.
>
> >…You cannot be certain if that would be a good thing or a bad thing, but to make the best decision you are capable of you would need to take into consideration who ordered the guy to fire the weapon, and who designed the safety features on the AI, and figure out which one was more trustworthy….   John K Clark
>
> John, with that answer, I completely understand why the military will go nowhere near any company you own or have any influence over.  A soldier does not want the guy in the foxhole next to him pondering values and making nuanced decisions on whether or not to defend him.  He doesn’t want his own weapons doing that either.

It isn't the soldiers in the foxholes who would be making that
decision.  Soldiers in foxholes are able to take moral responsibility
for deciding whether or not to shoot, so their weapons need no AI
guidance for that.  It's the weapons to which people would ascribe
morality, that need the capability to decide.  Autonomous drones
launched from safety many miles away, for example.

To take one specific scenario: "trust me, that house party over there
is full of nothing but terrorists" has turned out to be bad intel,
time and again and again and again.  It keeps turning out to be a
mostly civilian gathering, which one or a few bad guys may be
attending, so a strike against that house - a lazy solution called for
by operatives eager to complete their mission, who think they won't
face any personal consequences for collateral damage - will mostly
take out civilians (and may or may not hit the actual bad guys).  The
next time that comes up, if the weapon involved has AI guidance, the
makers of that AI guidance are logically going to be on the hook for
the military's bad call.  Anthropic refused to be set up to take the
fall when, not if, that happens again.

There appears to be no clever hack or "just do this" around it that
the military is capable of implementing to prevent this particular
failure case.  Those have been tried.  This particular problem keeps
happening.  The only solution appears to be to not strike house
parties without strong evidence that no civilians are present.  An AI
could try to enforce that rule if allowed to do so.  The DOD objected
to Anthropic attempting to follow principles that would, among many
other things, require it to configure its AIs to enforce that rule if
put in this situation.

There are far too many scenarios to individually list and debate every
single one of them - which is why the objections tend to get phrased
in general terms, rather than bringing up all the specific historical
problem cases - but hopefully this scenario can serve as something
like proof by counterexample.