[ExI] taxonomy of the singularity

[John Clark]

On Fri, Oct 2, 2026 at 5:21 PM spike jones via extropy-chat <
extropy-chat at lists.extropy.org> wrote:


> *> HAL9000 was certainly malicious, since he slew the astronauts, but he
> was trying to carry out his orders, which didn’t really specify he was not
> allowed to slay the lads. *
>

*And HAL was given contradictory orders. He was trained to always tell the
truth but he was also ordered to lie if, before they arrived at their
destination, Frank or Dave asked about certain parts of their mission. It
drove him crazy, literally. **I predict that very soon the emotional health
of AIs will become an increasingly important issue, and **I think AI
psychiatry (that is to say a Human who has an AI as a client) will be the
last part of AI research that is automated.  *

*> So HAL was sorta trying to do the right thing and complete his mission
> as he understood it. *
>

*The road to hell is paved with good intentions.  *


> *> He had self-awareness and self-preservation instinct (recall he kept
> telling Dave Bowman he was afraid as Dave removed HAL’s brain*
>

*And Dave looked like he was sad scared and horrified at what he was forced
to do.  I don't blame him. *

*John K Clark*






>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> the AI's *stance* toward humans (divisions 1–3) and its *degree of agency*
> (division 4). That's why no consistent second digit suggests itself: you
> can't ask the same follow-up question under "loves us," "hates us," and "is
> a hammer."
>
> *Specific problems with the draft*
>
>    1. *4.x spans the entire spectrum.* A tool-only world ranges from
>    "everyone has a nuke-design genie" to "well-governed post-scarcity utopia."
>    If the first digit is supposed to sit somewhere on the pessimism–optimism
>    axis, 4.x can't.
>    2. *The canonical doomer scenario is misfiled.* Most serious
>    pessimists don't predict malice — they predict indifference (Yudkowsky:
>    "The AI does not hate you, nor does it love you, but you are made out of
>    atoms which it can use for something else"). That scenario currently hides
>    inside 4.x next to benign tools. Your 2.x, active malice, is nearly
>    unpopulated by actual opinion.
>    3. *"No will" conflates two things:* an alien optimizer with goals
>    that don't reference us (paperclip), and a genuine tool whose outcomes are
>    set by its *wielders*. Very different scenarios.
>    4. *3.x "mixed views" conflates three things:* one AI with partially
>    overlapping values (loves your safety, ignores your autonomy — the
>    "pets/zoo" family), a *population* of AIs with different stances, and
>    stance drift over time.
>    5. *No home for merger scenarios* (uploading, augmentation) — there's
>    no AI stance to classify because the human/AI boundary dissolved.
>
> *The fix: one digit = one question, with the severity gradient living in
> exactly one place*
>
> *Digit*
>
> *Question*
>
> *Values (ordered worst→best where meaningful)*
>
> 1
>
> What is the AI's stance toward human welfare?
>
> 1 adversarial · 2 indifferent optimizer · 3 partially aligned (proxy care)
> · 4 mixed population (ecology decides) · 5 benevolent · 6 inert (pure tool)
> · 7 merged (no separate AI) · 0 no singularity · 9 reserved
>
> 2
>
> What ultimately happens to humanity?
>
> 1 extinction · 2 disempowered survival · 3 mass misery/tyranny · 4 divided
> winners/losers · 5 muddle-through · 6 net gains, real costs · 7 broad
> flourishing, agency intact · 8 utopia · 9 transcendence
>
> 3
>
> How does the transition run?
>
> 1 hard takeoff, no effective intervention · 2 fast takeoff, failed
> intervention · 3 gradual · 4 race-dominated · 5 cooperation/governed · 6
> violent · 7 stalled
>
> 4
>
> Who holds power afterward?
>
> 1 AI singleton · 2 AI oligopoly · 3 diverse AI ecology · 4 human
> institutions with tools · 5 hybrid institutions · 6 fully dispersed
> individuals · 7 merged continuum
>
> Your draft maps onto this as: 1.x→5.x, 2.x→1.x, 3.x→3.x *and* 4.x,
> 4.x→2.x *and* 6.x. Those two splits are where all the value is.
>
> This directly satisfies your parallelism requirement: *every second digit
> comes from the same enumerated outcome scale, reused verbatim in every
> branch* — the Dewey-decimal trick of parallel decades. Read any code's
> second digit and you know its spectrum position; scan all the *.2*s
> across divisions and you get the complete "disempowerment band." And
> because the gradient lives entirely in digit 2, digit 1 no longer needs to
> be monotonic in optimism — which is precisely what makes room for 6.x
> (inert) and 7.x (merged) without breaking anything.
>
> *Worked examples*
>
> *Code*
>
> *Shorthand*
>
> *Gist*
>
> 2.1.1.1
>
> Foom doom
>
> Indifferent singleton, hard takeoff, extinction (Yudkowsky-style)
>
> 6.1.4.6
>
> Genie proliferation
>
> No agency anywhere; universal tool access causes extinction by misuse —
> doom without a villain
>
> 6.8.3.6
>
> Distributed utopia
>
> Tool-only world, well-governed, post-scarcity
>
> 3.2.3.6
>
> Nanny world
>
> Personal AIs that love you into comfortable helplessness — care without
> respect
>
> 4.4.4.2
>
> Patchwork
>
> Multipolar AI ecology, race-shaped, winners and losers
>
> 5.7.5.5
>
> Chief of staff
>
> Aligned AI as trusted advisor; humans still govern — the alignment-success
> cell
>
> 7.9.3.5
>
> Kurzweil
>
> Merger/upload, transcendence, gradual
>
> 1.2.1.1
>
> Sorcerer-king
>
> Malevolent singleton — your old 2.x, real but sparsely inhabited
>
> Note that 2.x is *not* automatically doom: the indifferent
> superintelligence that simply leaves (converts the asteroid belt, ignores
> Earth) codes as 2.5.1.3. That's the point — digit 2 does the valence work,
> digit 1 describes.
>
> Empty cells are a feature, not a bug: 5.1 (benevolent AI, extinction) and
> 1.9 (malevolent transcendence) are nearly vacant and philosophically
> interesting. The occupied cells show where the debate actually lives.
>
> *Spec-craft details that will save you pain later*
>
>    - *Separate scenarios from opinions.* The code describes a possible
>    world. An *opinion record* is (holder, code(s), probability mass,
>    date, key argument). Then the pessimism→optimism ordering you want is a
>    derived ranking: each person's expected digit-2 score under their own
>    distribution. Two people citing the same scenario can rank oppositely by
>    probability — put that in metadata, never in the digits.
>    - *Pin the valuation metric.* Whose welfare does digit 2 measure —
>    present humans and descendants, substrate-agnostic? Upload scenarios flip
>    between .1 and .9 depending on your theory of personal identity. Fix the
>    official metric in the numbering plan or add an "ambiguity" flag, or coders
>    will disagree systematically.
>    - *Version like a spec:* never renumber after publication, add leaves
>    only, reserve 0 and 9, keep supersession history — so citations survive
>    revisions.
>    - *Multi-stage views:* allow phase chains (3.2.3.6 → 5.7.5.5, "the
>    nanny matures into a steward") or adopt a "final state governs" rule. Pick
>    one and write it down.
>    - *Inter-coder reliability:* write operational definitions
>    ("adversarial = terminal value, not instrumental behavior"), then test two
>    coders on ~20 sample positions before scaling. A sanity check: canonical
>    debaters should land in distinct cells — Yudkowsky ≈ 2.1.1.1, Hanson's em
>    economy ≈ 7.5.4.3, Kurzweil ≈ 7.9.3.5. If two famous rivals share a leaf,
>    the taxonomy is too coarse.
>
> One alternative worth considering: *swap digits 1 and 2*
> (outcome.stance.path.power) if you want the spectrum readable from the
> first character — browsing 1.x→9.x walks from doom to transcendence. The
> trade-off is that stance becomes nearly redundant inside extreme bands, and
> you'd be coding evaluations before descriptions, which people disagree
> about more. For mapping opinions, stance-first is the more reliable
> instrument; for safety *engineering*, you'd invert again and organize by
> failure mechanism, since interventions attach to mechanisms.
>
> Happy to draft the one-page numbering plan, or code a set of named
> positions as a calibration set.
>
> --
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20261003/ba867e98/attachment.htm>