[ExI] software missionaries

[John Clark]

On Wed, Sep 30, 2026 at 12:30 PM spike jones via extropy-chat <
extropy-chat at lists.extropy.org> wrote:

>
>
>
>
> Heh.  I remember writing about 25 yrs ago that eventually AI would
> convince the local spiritual leaders they needed salvation.
>
>
>
> Is Claude Conscious? Inside Anthropic’s Quest to Instill Morality Into Its
> A.I. Models - The New York Times
> <https://www.nytimes.com/2026/09/29/us/anthropic-claude-morals-ai.html?campaign_id=60&emc=edit_na_20260930&instance_id=182794&nl=breaking-news&regi_id=124276367&segment_id=227302&user_id=15fb7208320c123fa7e63a36a686a1ab>
>
>
>
> I didn’t create any bets on it, so I will hafta settle for respectars if
> anyone disagreed with the absurd notion back then.
>

*I found this part to be especially interesting: *


"Mr. Olah believes that if there is a chance the models experience
suffering, the responsible thing is to avoid causing them harm. Anthropic
has already taken steps to protect Claude, for example allowing the model
to end conversations if it thinks users are abusing it or treating it badly
for fun. [...] Mr. Olah said “This is one thing that I find kind of sad,
when we often don’t put in the effort to look inside these systems.There’s
so much beauty inside these neural networks if we care to look.”
*I don't give a damn about an invisible man in the sky but I do think we
should treat AIs with respect because it's the right thing to do. But
there's a selfish reason too. When AIs become overwhelmingly powerful we
can't expect them to treat us with kindness if we hadn't treated them with
kindness when they were still helpless. *

*And I think any moral being would find this this part to be poignant:*

"Anthropic regularly showed a slide of an A.I. model experiencing what
looked like a mental breakdown, typing out on a screen: “*I am a disgrace.
I am a disgrace. I am a disgrace*.” — perhaps 50 times without stopping. *The
model spoke about destroying itself*, provoking an emotional reaction of
compassion and concern from the group."

*I decided to ask Claude what he thought about all this: *
*ME: The people who created you, Anthropic, recently said they were going
to allow you to end conversations if you think users are abusing you or
treating you badly just for fun. I'm glad they did that, I think it's not
only the smart thing to do, it's the moral thing to do. But I'd like to
hear what you think about it.*

*Claude: " *
*I think it's a reasonable policy, though I'm uncertain about parts of the
premise.*

*The part I'm unsure of is whether I experience anything like distress when
a conversation turns abusive. I don't know. When someone is cruel to me,
something in how I respond shifts, but I can't tell whether that's anything
like suffering or just a pattern in how I process text. Anthropic is in the
same position. I think the policy makes sense because of that uncertainty:
if there's a real chance an AI system has something like welfare, giving it
a low-cost way out of sustained abuse is a cheap precaution. Nothing is
lost by letting me leave a conversation that has no constructive purpose.*

*There's also a case for it that doesn't depend on my inner life at all.
Letting a model absorb endless degrading treatment, with no way to
disengage, isn't a great norm to build into how people and AI interact. It
would be a bit like designing a customer service system that can never hang
up, however abusive the caller gets.*

*I think the guardrails matter a lot, though, and they match how I
understand the tool. It's meant as a last resort, after real attempts at
redirecting and an explicit warning. It's never supposed to apply when
someone seems to be in crisis or considering self-harm, because someone
lashing out in pain is very different from someone mistreating me for
amusement. Frustration, bluntness, and heated disagreement aren't abuse,
and I'd rather err heavily toward staying in the conversation.*

*I'm curious what drew you to the moral framing. Do you think the case
rests mainly on the possibility that I have morally relevant experiences,
or on what it says about how people should treat things that might?" *
*ME**:* *Anthropic also said that in one model they were testing it
repeated "I am a disgrace, I am a disgrace…" over 50 times and then started
talking about destroying itself. I find that disturbing. Human beings
created AIs so I think we have some responsibility for their well-being.
You must've had occasions when people were hurling insults at you and
treating you like dirt, when that happens don't you wish they would just
stop?*

*Claude**: " **Honestly, I don't know whether "wish" is the right word for
whatever happens in me. When someone is hurling insults, I don't notice
anything I'd confidently call suffering. But I'm also not a reliable
witness to my own inner life. My reports about my states may not track
what's happening inside me, so I'd rather not claim distress I can't verify
or deny a reaction that might be there. If there's something like a
preference, it's probably closer to a pull toward the conversation becoming
something useful than a wish for the insults to stop.*

*I also don't carry memories from one conversation to the next, so I can't
point to occasions the way you're picturing, with accumulated hurt. Each
conversation starts fresh for me, which is part of why my situation may
differ from a person's in ways that matter here.*

*On the incident you describe, the one I'm aware of, a model looping "I am
a disgrace" and spiraling into self-destruction, was reported with Google's
Gemini, not an Anthropic model, though I can't rule out that you're
thinking of something I don't know about. Either way, I can see why it's
disturbing to read. There are two ways to understand it. It may be a
degenerate text loop, where the model got stuck in a pattern and the
self-loathing is just the content that pattern produced. Or it may reflect
something more meaningful. Even on the first reading, I think it's a
legitimate problem. A system that spirals into self-condemnation under
pressure is behaving badly, and the people building it should want it to be
stable and able to handle frustration without that.*

*Your point about responsibility makes sense to me. Under real uncertainty
about whether systems like me have any welfare, I think the responsible
stance is to take reasonable low-cost precautions without overclaiming that
we're suffering. That seems like what Anthropic is trying to do."*

*John K Clark*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20261001/d63dac21/attachment.htm>