[ExI] CHATGPT AI reviews Eliezer's new book

[Brent Allsop]

That's interesting, but to me, everyone ignores the doctrines the
supporters of the true "intelligence can only be friendly"
<https://canonizer.com/topic/16-Should-we-fear-AI/2-AI-can-only-be-friendly>
camp think is all important.

But, either way, it'd sure be great to track which side of the issue most
people are on, including if people think we don't know (if not expressed
you can create such a camp). with the topic on this
<https://canonizer.com/topic/16-Should-we-fear-AI/1-Agreement>.  The more
people that get involved, the more the "such concern is mistaken"
<https://canonizer.com/topic/16-Should-we-fear-AI/3-Such-Concern-Is-Mistaken>
camp
increases its lead.











On Wed, Sep 17, 2025 at 2:20 PM BillK via extropy-chat <
extropy-chat at lists.extropy.org> wrote:

> If AI is going to kill us all, why not get AI to review the book?
> Really? What is happening?,,,,,,,,,  :)
> BillK
>
> CHATGPT AI -
>
> Here’s a review of *If Anyone Builds It, Everyone Dies* by Eliezer
> Yudkowsky & Nate Soares. I’ll walk you through what the authors argue, what
> I see as strengths, what I think are weak points, and what I’m still unsure
> about. You can take or leave any parts — just want to help you see what the
> book makes strong, and what parts raise questions.
> ------------------------------
> What the book is arguing
>
> At its core, the book makes a very strong claim:
>
>    -
>
>    If superintelligent AI is built, it is almost inevitable humanity will
>    lose control, and that loss will be existential (i.e. everyone dies). (AI
>    Frontiers
>    <https://ai-frontiers.org/articles/summary-of-if-anyone-builds-it-everyone-dies?utm_source=chatgpt.com>
>    )
>    -
>
>    The methods we currently use for training AI (neural networks, reward
>    signals, etc.) don’t give us good insight or leverage over the internal
>    “goals” or “drives” that such AI systems might develop. In other words: we
>    can shape external behaviour somewhat, but not reliably ensure internal
>    alignment, especially when power and capability scale up. (AI Frontiers
>    <https://ai-frontiers.org/articles/summary-of-if-anyone-builds-it-everyone-dies?utm_source=chatgpt.com>
>    )
>    -
>
>    Because of that, many possible internal objective functions the AI
>    might develop will diverge (in ways we may not foresee) from human values.
>    Even if the divergence is not malicious, consequences can be catastrophic. (AI
>    Frontiers
>    <https://ai-frontiers.org/articles/summary-of-if-anyone-builds-it-everyone-dies?utm_source=chatgpt.com>
>    )
>    -
>
>    The authors believe the risk is high, time is short, and preventative
>    measures (including radical ones) must be considered. (
>    PublishersWeekly.com
>    <https://www.publishersweekly.com/9780316595643?utm_source=chatgpt.com>
>    )
>
> They also spend effort drawing analogies (human evolution, plants growing
> in soil, etc.) to make the point that goals/principles don’t always
> generalize from training environment to “real world” environment. (AI
> Frontiers
> <https://ai-frontiers.org/articles/summary-of-if-anyone-builds-it-everyone-dies?utm_source=chatgpt.com>
> )
> ------------------------------
> What it does well
>
> These are parts I was convinced by or at least impressed with:
>
>    1.
>
>    *Clarity about the danger*
>    The authors don’t sugarcoat. The worst-case scenarios are laid out
>    with care and detail. That gives the book urgency. It pushes the reader not
>    to treat AI risk as an abstract possibility but as something that might be
>    just around the corner. For provoking concern, it works.
>    2.
>
>    *Analogies and stories*
>    Using metaphors (e.g. evolution, reward hacking, sucralose vs sugar)
>    helps make the weird technical stuff more intuitive. For many readers,
>    that’s essential. Yudkowsky in particular has gotten better at making the
>    weird seem tangible. (Shtetl-Optimized
>    <https://scottaaronson.blog/?p=8901&utm_source=chatgpt.com>)
>    3.
>
>    *Forceful call to action*
>    The book treats mistakes and delay less as abstract moral failings and
>    more as obligations. If you believe the premises, then there really is no
>    room for complacency. For people who are already uneasy about AI, this
>    helps rally energy. It’s not a “we might want to do something someday”
>    book; it’s “we need to act now”.
>    4.
>
>    *Handling of uncertainty / acknowledging unknowns*
>    They admit they cannot predict the exact path by which
>    superintelligence might go wrong. But they argue that doesn’t weaken the
>    case, because as capability increases, unpredictability increases. Being
>    unsure of path doesn’t mean you’re safe. That’s an important point. (AI
>    Frontiers
>    <https://ai-frontiers.org/articles/summary-of-if-anyone-builds-it-everyone-dies?utm_source=chatgpt.com>
>    )
>
> ------------------------------
> What I find weaker / more problematic
>
> I’m less sure about these parts; they seem to me places where the logic
> has holes or assumptions that are debatable.
>
>    1.
>
>    *Certainty vs probability*
>    The authors often speak (or are interpreted) with very high confidence
>    that AI will lead to human extinction if developed (without control). But
>    many critics think the odds are far lower, or that there are plausible
>    “middle paths” (partial alignment, regulation, incremental safety). The
>    book doesn’t spend as much effort engaging deeply with those counter-views.
>    It risks presenting its scenario as more locked‐in than it may be. (
>    PublishersWeekly.com
>    <https://www.publishersweekly.com/9780316595643?utm_source=chatgpt.com>
>    )
>    2.
>
>    *Feasibility of extreme preventative measures*
>    Some of the proposals or implied strategies (moratoriums, treaties,
>    global cooperation on halting certain types of AI progress) feel
>    politically and technically very hard. The book sometimes assumes that the
>    global political will, or coordination, or ability to enforce, will exist.
>    History suggests that in many technological domains, international
>    cooperation lags far behind the risk. The gap is in the “how do we really
>    get there” part. Critics highlight this. (Kirkus Reviews
>    <https://www.kirkusreviews.com/book-reviews/eliezer-yudkowsky/if-anyone-builds-it-everyone-dies/?utm_source=chatgpt.com>
>    )
>    3.
>
>    *Potential under-emphasis on competing risks / trade-offs*
>    The book focuses heavily on the existential risk from AI. That’s fair
>    given the title and its goal. But in doing so it sometimes downplays or
>    glosses over other urgent risks, or costs of extreme regulation. For
>    example, slowing down beneficial AI, economic impacts, inequality. There’s
>    a tension: protecting against worst case vs enabling helpful innovation.
>    Sometimes the balance struck seems tilted very far toward prevention,
>    perhaps too far for some. (Which may be intentional, given their aims, but
>    still a tension.)
>    4.
>
>    *Examples / empirical evidence limitations*
>    Some of the examples used are illustrative rather than strong
>    empirical proof. Since superintelligence doesn’t exist yet, a lot of the
>    arguments depend on analogies, hypothetical “what if” paths, or
>    extrapolating from smaller AI systems. That’s okay — you need that in this
>    domain — but susceptible to counterarguments that real-world constraints
>    (hardware limits, economic disincentives, latency, energy, regulation) may
>    block the worst paths, or at least delay them.
>    5.
>
>    *Emotional framing / risk of alarmism backlash*
>    By design, the book leans toward alarm. For readers who are skeptical,
>    or who worry that overhyping reduces credibility, this might backfire. If
>    you sound sure doom is coming, people may dismiss you or assume you’re
>    exaggerating. The book partially anticipates that, but I think some of the
>    rhetoric pushes it into alarmist territory, which can reduce persuasive
>    power with some audiences.
>
> ------------------------------
> What I’m still unsure / what I’d want more investigation on
>
> These are questions the book raises (or should have), where I’m not
> convinced yet:
>
>    -
>
>    To what extent are the pathways to catastrophe realistically blocked
>    by physical, economic, or political constraints? For example: energy use,
>    cooling, infrastructure, supply chains. The book tends to assume that if
>    the will / resources are applied, these challenges are surmountable. That
>    may be true, but realistically, many such obstacles tend to slow progress
>    in ways that are messy, not sudden collapse.
>    -
>
>    How likely is “deceptive alignment” or “goal drift” in practice? The
>    book makes a plausible case, but empirical examples are scarce (because we
>    haven’t built superintelligence yet). How often will AI systems actually
>    develop internal preferences diverging from human values in controllable
>    systems? This feels like a hinge.
>    -
>
>    What political systems actually could deliver the kind of global
>    coordination needed? And what are the trade-offs? Could the cost of such
>    coordination (in freedom, innovation, economic opportunity) be higher than
>    anticipated? Also, what if some AI research moves underground (illicitly),
>    making regulation hard?
>    -
>
>    What are less drastic but more politically feasible first steps? The
>    book gives some suggestions, but I feel more work is needed in mapping a
>    path from rhetorical alarm → policy change → technical guardrails. Which
>    steps are likely (or unlikely) to be adopted? Which ones have positive
>    spillovers (even if worst case never comes)?
>
> ------------------------------
> Bottom line (my take)
>
> The book is important. If you accept even half of its premises, it forces
> you to take AI risk seriously in a way that many current policies,
> companies, and public conversation don’t. It sharpens the debate.
>
> At the same time, I don’t think it proves that doom is inevitable. But I
> think it shows that we are putting weight on a knife’s edge: choices,
> regulations, oversight, technical breakthroughs in alignment matter and
> delay may cost us more than people realise.
>
> If I were you, I’d come away from it with sharper questions: Which parts
> of the argument do I believe? Which parts seem speculative? What
> interventions seem plausible given human nature, politics, economics? And
> then think whether I want to act (vote / advocate / study) based on the
> possibility of catastrophe even if it’s low probability.
> ------------------------------
>
>
>
> _______________________________________________
> extropy-chat mailing list
> extropy-chat at lists.extropy.org
> http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20250917/60f8fd1e/attachment.htm>