[ExI] Pope Leo and AI

[Keith Henson]

Simon

This is such a good description of a solution to the problem that I am
forwarding it to the three AI researchers I know and the Extropian list.
I might send it elsewhere. It is consistent with the AI character I wrote
into the Clinic Seed story 20 years ago.

Keith


On Thu, Jun 11, 2026 at 6:18 PM Simon Quellen Field AB6NY <
simon.field at gmail.com> wrote:

> I am not so interested in replacing humanity.
> A machine that can reason is useful.
> One that won't let me shut it off is not. Let's not make those.
>
> We're making tools. We aren't having children.
>
> We have so many exceptions to what you call "moral rights" that the term
> has almost no meaning. We give our pets rights. If you shoot my dog, you
> can go to jail. But we also take those rights away. If I give my dog to the
> Humane Society, they will kill it without any legal penalty. We kill
> people. We train people to kill people. A large part of the world economy
> is involved in killing people. But if you shot me, you would go to jail.
> But I can get a doctor to kill me.
>
> You can't claim that there are absolute "moral rights" and still allow all
> these exceptions. We decide what we tolerate and what we don't. We can
> decide that it is always OK to turn off my computer.
>
> If a foreign country has an agenda that differs from ours, we often go to
> war over those differences. They may want our resources, our land, or they
> might just not like our religion. If we allow our computer programs that do
> reasoning for us to have agendas, those agendas might differ from ours. We
> should not create machines that want things, or get sad, or get angry, or
> can have their feelings hurt. That would be dangerous, for no good reason.
>
>
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> Virus-free.www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> <#m_8527086621778811963_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> On Thu, Jun 11, 2026 at 5:45 PM 'William Arnett' via Inventor's Lunch <
> inventors-lunch at googlegroups.com> wrote:
>
>> When people anthropomorphize LLMs
>>
>>
>> I am NOT anthropomorphizing. I am asking questions about what it is that
>> we value so much in humanity and whether we are necessarily unique in that
>> regard. There have long been simple answers: We are made in the image of
>> God. Might makes right. We are the pinnacle of evolution... All bullshit.
>> I’m hoping (not asserting) that we can develop some better answers from
>> building and studying AIs.
>>
>> We’ve already made a little progress. Years ago, many people thought that
>> the Turning Test was a sufficient condition for moral worth. That is no
>> longer a widely held opinion (though I’m not so sure). It’s a negative
>> result but better than nothing.
>>
>> , they confuse intelligence with intent.
>> LLMs have no ambition. They have no desire. Self-preservation is
>> something that comes from Darwinian survival of the fittest. LLM's did not
>> evolve this way. They don't care when you turn them off.
>>
>>
>> Those are all unjustified assertions. But even if they are true, there’s
>> no reason to believe the biological evolution is so special that the kind
>> of evolution that produces LLMs might not also create those qualities.
>>
>> We already have superintelligence. People still play chess, even though
>> computers can beat the best of them. Protein-folding is now done much
>> faster by computer programs, but the people who used to do that haven't
>> lost their jobs. We still have tax preparers in the age of TurboTax. The
>> number of carpenters did not drop when we invented power tools. We just
>> made houses faster and hired more carpenters. LLMs are power tools.
>>
>>
>> Yes, but are we at the end game already? There’s no a lot to be improved
>> with my power drill. But LLMs are getting better very rapidly. How can you
>> be so sure that they will never be X or Y or Z?
>>
>> We did lose jobs like "typist". Now everyone is a typist. Now everyone is
>> a typesetter. Soon (if not already), everyone will be a graphic designer.
>> The skills those people have turned out to be useful elsewhere, and they
>> got new jobs, or did the job faster and better using the tools, making more
>> money.
>>
>>
>> There’s a long history in this regard. And it’s usually wrong to say
>> “this time is different”. But there ARE significant differences this time.
>> The Industrial Revolution took a century. AI is coming for your job next
>> year. Past tech advances always left room for people to switch to other
>> jobs. But if AI can do everything that’s not going to work.
>>
>> Governments are already creating laws that require models to be
>> government tested for safety before being released to the public.
>>
>>
>> That’s not going to work.
>>
>> Science fiction is not fortune-telling. It is a forewarning of possible
>> dangers, giving us time to think of possible consequences and take action.
>>
>>
>> Yes, exactly. And IMHO in the best cases it provides a hopeful future to
>> strive for.
>>
>> Hinton thinks LLMs are already conscious. Has anyone seen evidence that
>> they have taken over and subjugated us?
>>
>>
>> << If they have taken over would you know? They’re so smart that they can
>> control us without our knowledge. And the government is hiding the
>> evidence. The Illuminati is an AI. >>
>>
>> Why do you assume that a conscious AI is necessarily malevolent?  Is it
>> not a better strategy to work to make sure it’s beneficial?
>>
>> On Jun 11, 2026, at 3:33 PM, 'Steve Pucci' via Inventor's Lunch <
>> inventors-lunch at googlegroups.com> wrote:
>>
>> Well, we could just apply Betteridge's Law to the headline. :)
>>
>>
>> Headline: “Will the Sun rise tomorrow?”, “No” is not the right answer. In
>> this case, I think the question is justified in that it’s a statement about
>> the future which the article (correctly) asserts is very uncertain.
>>
>> The people saying that we should worry about this are postulating an
>> extraordinary claim: that somehow, through no mechanism that anyone is even
>> attempting to explain, moral rights can arise from the complexity of a
>> machine.
>>
>>
>> They (and I) are claiming only that this MIGHT happen. Given the paucity
>> of real knowledge in this domain I think that’s at the very least something
>> worth paying attention to. The evidence we have at present is ourselves and
>> IMHO we are just complex machines, too.
>>
>> Re Anil Seth conceding there is no "knockdown argument that consciousness
>> requires a biological substrate":
>> * Let's avoid conflating "reasoning ability" with "consciousness" and
>> "consciousness" with "moral rights".
>>
>>
>> OK. But what do we base moral rights upon? Right now it’s essentially
>> possession of human DNA. But I’ve never been satisfied with that.  And What
>> is “consciousness” anyway? No one has a clue.
>>
>> * We don't need to prove that. All we need to prove is that moral rights
>> can't arise with current AIs or ones we create based on the same principles.
>>
>>
>> And we have most certainly NOT proved that moral rights can not arise.
>> Proving a negative is tough :-) But more practically, we have little
>> evidence either way. Especially since we don’t even know that the words
>> mean.
>>
>> Re the evidence of introspection and emotional states inside model states:
>> * An internal state that represents happiness is again a map, not the
>> territory.
>>
>>
>> This is the “qualia” argument again. It has never made sense to me. There
>> is no mysterious substance that *is* happiness as opposed to *representing*
>> it. The two are the same thing.
>>
>> I can maintain an internal state inside Eliza if I want to -- should we
>> grant moral rights to Eliza if I do that? The fact that it arises
>> "spontaneously" is no argument at all, it seems to me: Either the state has
>> meaning by itself or it doesn't, regardless of how mysteriously the state
>> was constructed (and it seems to me it's not at all mysterious; emotional
>> states and introspection are subsets of many abstractions that LLMs have to
>> make to reason properly, particularly when they need to check their own
>> output)
>>
>> Re Anthropic's constitution:
>> * We've discussed this before; we shouldn't assume that because some
>> smart people believe a thing to be true, it is necessarily true. Whatever
>> religion you have or don't have, many smart people believe something
>> completely different. As models grow more complex and useful, the
>> temptation to anthropomorphize grows, even among smart people.
>>
>>
>> Maybe this time anthropomorphism isn’t wrong?
>>
>> I wouldn't care so much about this if there weren't large unambiguous
>> ethical concerns about AI that have nothing to do with giving them moral
>> rights, and I'm worried that introducing this discusion will distract a
>> fickle public.
>>
>>
>> In that much we can agree. Except as a wannabe philosopher and hopelessly
>> ineffective politician, I find the bigger issues more interesting. And on
>> both levels, I’m worried that just when we need careful thought and an
>> informed public debate we have a totally broken political environment.
>>
>>
>>
>> On Jun 11, 2026, at 3:46 PM, Simon Quellen Field AB6NY <
>> simon.field at gmail.com> wrote:
>>
>> I'm with Steve.
>> The machines are designed to mimic people, so it is easy to
>> anthropomorphize them.
>> Plus all of the science fiction that makes them characters basically
>> makes them human.
>> In movies, they are played or voiced by humans.
>>
>>
>> To the extent that fictional AIs are portrayed as human-like that’s
>> mostly for dramatic purposes. For humans to relate to the story it needs
>> characters that the reader can understand. The intent is to
>> anthropomorphize. But there are plenty of sci-fi AIs that aren’t even close
>> to human. (Eg. Iain Bank’s Culture novels.)
>>
>> I have yet to hear a definition of consciousness.
>>
>>
>> Maybe mechanistic interpretation will provide some insight?
>>
>> But I can imagine a reasoning machine that has introspection, and still
>> doesn't care if I turn it off, insult it, or set fire to it.
>>
>>
>> Sure. And I can imagine one that does care. Imagination can be a useful
>> guide but what we really need is facts.
>>
>> I might go so far as to say that if a machine fears being turned off, we
>> should turn it off right away.
>> Then perhaps set fire to it.
>>
>>
>> I’m not afraid of a future where humans are no longer the masters of the
>> universe. We’re doing such an abysmal job of it now that a change has at
>> least an even chance of being better.  Of course, IMHO, it’s very unlikely
>> that we actually are the masters of the universe. In all those billions and
>> billions of galaxies there is very likely something out there vastly more
>> powerful than we are. Our lofty opinion of ourselves is mostly based on
>> ignorance. AI is just bringing the issue to the fore without having to wait
>> for the aliens to arrive.
>>
>>
>> Bill Arnett  〜  bill at arnett.us.com
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Inventor's Lunch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to inventors-lunch+unsubscribe at googlegroups.com.
>> To view this discussion visit
>> https://groups.google.com/d/msgid/inventors-lunch/922B53F0-D62D-417A-A9A7-53C0ED6E2627%40mac.com
>> <https://groups.google.com/d/msgid/inventors-lunch/922B53F0-D62D-417A-A9A7-53C0ED6E2627%40mac.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Inventor's Lunch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to inventors-lunch+unsubscribe at googlegroups.com.
> To view this discussion visit
> https://groups.google.com/d/msgid/inventors-lunch/CAA0yOM5WF8R_teW-RN6ppKxLiZfaXh0qfJzgn385dmerieYHRw%40mail.gmail.com
> <https://groups.google.com/d/msgid/inventors-lunch/CAA0yOM5WF8R_teW-RN6ppKxLiZfaXh0qfJzgn385dmerieYHRw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20260611/ea044b46/attachment-0001.htm>