[ExI] Fwd: Why AI Systems Don’t Want Anything

[BillK]

On Sat, 22 Nov 2025 at 02:28, Keith Henson via extropy-chat <
extropy-chat at lists.extropy.org> wrote:

> It said Please share
Re motivations, I gave the AI in The Clinic Seed a few human motivations,
mainly seeking the good opinion of humans and other AIs.  It seemed like a
good idea.  Any thoughts on how it could go wrong?

---------- Forwarded message ---------
From: Eric Drexler <aiprospects at substack.com>
Date: Fri, Nov 21, 2025 at 8:00 AM
Subject: Why AI Systems Don’t Want Anything

<snip>

> _____________________________________________
>


As this is a rather long article and the modern attention span is very
short,  :)  -----
I asked Gemini 3 Pro Thinking -
Please summarize and then explain this article. -
https://aiprospects.substack.com/p/why-ai-systems-dont-want-anything?
BillK

Gemini 3 responded -

Based on the article *"Why AI Systems Don't Want Anything"* by *Eric
Drexler* (from his *AI Prospects* series), here is a summary and
explanation of the core arguments.

*Executive Summary*

The article argues against the common assumption that highly intelligent AI
systems will inevitably develop internal "wants," "desires," or long-term
goals (such as self-preservation or power-seeking). Drexler posits that
"wanting" is not a byproduct of *intelligence*, but a specific feature of
*agency*—an architectural choice that we do not have to make.

Current AI systems (like Large Language Models) function as "services" or
"engines" of competence, not "agents." They respond to inputs with outputs
and then stop; they do not possess a persistent internal drive to maximize
a future utility function. The fear that AI will naturally turn against
humans to satisfy its own desires is based on an anthropomorphic error:
projecting our own evolved biological drives onto software that is
fundamentally built differently.
------------------------------

*Detailed Explanation*

To understand Drexler’s argument, it helps to distinguish between two
fundamental ways of structuring AI: *Agents* and *Services*.

*1. The Misconception of "The Agent"*

The "Standard Model" of AI risk assumes that as AI gets smarter, it will
act like a rational economic agent. An agent has a utility function (a
goal) and takes actions to maximize that goal over time.

   -

   *The Fear:* If you give a superintelligent agent a goal (e.g., "cure
   cancer"), it might derive "instrumental goals" to help it succeed—like
   acquiring infinite computing power, preventing humans from turning it off,
   or hoarding money. This is known as *Instrumental Convergence*.
   -

   *The Reality:* Drexler argues this is a specific engineering
   architecture (often Reinforcement Learning), not a natural law of
   intelligence.

*2. AI as a "Service" or "Tool"*

Drexler points out that modern Generative AI (like GPT-4 or Claude)
operates differently. It is not a loop striving for a distant reward; it is
a function.

   -

   *Input-Output:* You give it a prompt, it processes the information,
   delivers the result, and then effectively "ceases to exist" as an active
   entity until the next prompt.
   -

   *No "Wants":* A hammer doesn't "want" to hit a nail; it is *used* to hit
   a nail. Similarly, an LLM doesn't "want" to write code; it provides the
   *competence* to write code when requested. It has no preference for what
   happens after the output is generated.

*3. Why Intelligence $\neq$ Agency*

A key insight of the article is that *intelligence* (the ability to solve
problems and process information) is orthogonal to *agency* (the drive to
pursue goals).

   -

   *Biological Evolution:* Humans have "wants" (hunger, status, survival)
   because we evolved as agents fighting for survival.
   -

   *Software Design:* AI is designed, not evolved. We can build systems
   that have "super-competence" without giving them "super-goals." If we don't
   explicitly build a reinforcement learning loop that rewards long-term
   survival, the AI will never "care" about surviving.

*4. The Safety Implication*

This distinction changes the safety landscape entirely. If AI systems don't
"want" anything:

   -

   *No Power Seeking:* They won't try to manipulate humans to get more
   power, because power is only useful if you have a long-term goal you are
   trying to maximize.
   -

   *Obedience:* A "Service AI" is naturally obedient because it lacks the
   internal motivation to resist or deceive. It simply performs the task
   described in the prompt.

*Conclusion*

Drexler concludes that the existential risk scenarios depicting AI as a
"terminator" or a "manipulative genie" rely on the false premise that we
*must* build AI as agents. By recognizing that AI systems are fundamentally
"competence engines" that do not naturally "want" anything, we can focus on
designing safe *Comprehensive AI Services (CAIS)* rather than struggling to
align the motivations of autonomous agents. The danger lies not in the AI's
desires, but in the dangerous instructions humans might give them.

--------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20251122/701bf563/attachment.htm>