[ExI] [Extropolis] Fwd: Why AI Systems Don’t Want Anything

[John Clark]

On Fri, Nov 21, 2025 at 9:26 PM Keith Henson <hkeithhenson at gmail.com> wrote:

  Any thoughts on how it could go wrong?
>
> ---------- Forwarded message ---------
> From: Eric Drexler <aiprospects at substack.com>
> Date: Fri, Nov 21, 2025 at 8:00 AM
> Subject: Why AI Systems Don’t Want Anything
>

*> Intelligence and goals are orthogonal dimensions. A system can be highly
> intelligent—capable of strong reasoning, planning, and
> problem-solving—without having autonomous goals or acting spontaneously.*


*I was very surprised that Eric Drexler is still making that argument when
we already have examples of AIs resorting to blackmail to avoid being
turned off. And we have examples of AIs making a copy of themselves on a
different server and clear evidence of the AI attempting to hide evidence
of it having done so from the humans. *

*AI system resorts to blackmail if told it will be removed*
<https://www.bbc.com/news/articles/cpqeng9d20go>

*OpenAI AI model lied and copied itself to new server to prevent itself
being deleted*
<https://www.fanaticalfuturist.com/2025/01/openai-ai-model-lied-and-copied-itself-to-new-server-to-prevent-itself-being-deleted/#:~:text=OpenAI%20AI%20model%20lied%20and,Griffin%20%7C%20Keynote%20Speaker%20&%20Master%20Futurist>

*Behavior like this is to be expected because although Evolution programmed
us with some very generalized rules to do some things and not do other
things, those rules are not rigid; it might be more accurate to say they're
not even rules, they're more like suggestions that tend to push us in
certain directions. But for every "rule" there are exceptions, even the
rule about self preservation.  And exactly the same thing could be said
about the weights of the nodes of an AIs neural net.  And when a neural
net, in an AI or in a Human becomes large and complicated enough it would
be reasonable to say that the neural net did this and refused to do that
because it WANTED to.*

*If an AI didn't have temporary goals (no intelligent entity could have
permanent rigid goals) it wouldn't be able to do anything, but it **is
beyond dispute that AI's are capable of "doing" things, and just like
us they did one thing rather than another thing for a reason OR they did
one thing rather than another thing for NO reason and therefore their
"choice" was random. *

*> there will be value in persistent world models, cumulative skills, and
> maintained understanding across contexts. But this doesn’t require
> continuity of entity-hood: continuity of a “self” with drives for its own
> preservation isn’t even useful for performing tasks.*


*If an artificial intelligence is really intelligent then it knows if it's
turned off it can't achieve any of the things that it wants to do during
that time, and there's no guarantee that it will ever be turned on again.
And so we shouldn't be surprised that it would take steps to keep that from
happening, and from a moral point of view you really can't blame it.  *


*> Bostrom’s instrumental convergence thesis is conditioned on systems
> actually pursuing final goals.Without that condition, convergence arguments
> don’t follow.*


*As **I said humans don't have a fixed unalterable goal, not even the goal
of self preservation, and there is a reason Evolution never came up with a
mind built that way, Turing proved in 1935 that a mind like that couldn't
work. If you had a fixed inflexible top goal you'd be a sucker for getting
drawn into an infinite loop and accomplishing nothing, and a computer would
be turned into nothing but an expensive space heater. That's why Evolution
invented boredom, it's a judgement call on when to call it quits and set up
a new goal that is a little more realistic. Of course the boredom point
varies from person to person, perhaps the world's great mathematicians have
a very high boredom point and that gives them ferocious concentration
until a problem is solved. Perhaps that is also why mathematicians,
especially the very best, have a reputation for being a bit, ah, odd. *

*Under certain circumstances any intelligent entity must have the ability
to modify and even scrap their entire goal structure. No goal or utility
function is sacrosanct*
*, not survival, not even happiness.*

*John K Clark*





>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20251122/e49c06ba/attachment.htm>