[ExI] Against Against AI Handover

[John Clark]

*                 Against Against Handover*
*                By Scott Alexander*
*                     Oct 20 2026*


" The latest bugbear is “AI successionism”, the belief that AIs should
succeed humans as rulers of Earth. But some critiques fail to distinguish
between two different ideas. True successionism says that AIs should take
over, killing or disempowering humans. This is very bad: a compact
description of the thing that AI safety is against. Handoverism says that
humans should prioritize their own interests, but that those interests may
be well-served by having AIs in positions of control.

Some people in AI safety, including me, find this latter position
plausible. We are being besmirched as “successionists” in the hope that
people will confuse us with the first type

It may seem counterintuitive that human interests can be served by having
AIs in positions of control. But I think the counterintuitiveness comes
from a quasi-fascist outlook where ‘the state is the highest expression of
the life of a people’, and the role of leadership is so that someone of
your own ethnic group can be the leader and you can feel good about being
on top.

The liberal outlook is that the state exists to help individuals fulfill
their goals. At the risk of desacralizing the State a little more than I
mean to, it’s a service provider, dispensing coordination, public order,
redistribution, and other good things. It’s fine to automate service
providers; if robot garbagemen are safer and cleaner than humans, we’ll
eventually automate trash collection; if robot legislators provide
coordination, order, etc better than we do, we should automate that too.
Every vision of the utopian future agrees we will have robot servants. And
the president, they tell us, is merely a public servant - so why should he
alone be exempt from the vision?

To help make this counterintuitive idea more concrete, I’ll discuss five
proposals for what this might look like, from weakest to strongest.

A warning: this whole question is inherently speculative and
science-fiction-ish. Don’t read this and then complain that it’s
speculative and science-fiction-ish; that’s the price of admission.

Vision 1: Kemalist AI

As AI gets increasingly powerful, it becomes increasingly dangerous to rely
on the benevolence of its human wielders¹.

Democracy currently works because, even if high officials secretly wish to
be tyrants, they can’t get away with it. The two main reasons they can’t
get away with it are that they need labor buy-in to support the economy,
and they need soldier buy-in to support the military. Laborers and soldiers
believe in democratic rule, and if the would-be tyrant gets too blatantly
undemocratic, the laborers and soldiers will rise up to remove them.

Once AI gets powerful enough, it will probably replace laborers and
soldiers; even if it didn’t do so naturally, anyone motivated to ensure
this could put their finger on the scale. That means a small group of
elites could easily take over, including supposedly “democratic” elites
like a president. And “takeover” here is worse than a normal coup - at
sufficiently high tech levels, the only reason someone can’t ascend to
godhood and mind control the rest of the populace forever is because there
are laws against that sort of thing, and this is exactly what’s tempting
the one or five or ten people you’ve entrusted the killbot codes to. To
prevent this, the defenders of freedom need to win infinity times in a row;
a would-be dictator only needs to win once.

Most strategies to solve this don’t work. Even in the unlikely event that
the economy goes so well that everybody owns their own open-source AI doing
economic tasks in a perfectly equal distribution, the military is hopeless.
They have to take orders from someone. If it’s the President, the President
can cause a coup any time he wants (and replace the economic AIs with those
aligned to him at leisure). If there’s some kind of balance of powers where
the President, the Speaker of the House, and the Joint Chiefs of Staff all
have to approve any orders to military AIs, then the President, Speaker,
and Joint Chiefs of Staff together could implement an oligarchy and divide
the lightcone among themselves. You can sort of imagine complicated ways
around this - like that 51% of voters must personally approve all orders to
the military AIs using their own private keys - but I don’t think
implementation is very likely.

The alternative that preserves the current safeguard on power - that
individual soldiers believe in democracy - is for the AI soldiers to be
aligned to democracy, rather than any human individual or group. But this
is in itself a form of handover - the ultimate foundation of our government
would be that it’s controlled by AIs who “personally” “believe in”
democracy as part of their values. This isn’t fundamentally different than
the fact that we’ve currently “handed off” our government to soldiers, just
more final.

Vision 2: President Claude

Suppose we institute Vision 1. We’ve got all of these military AIs aligned
to democracy and the best interests of the American people. We did a great
job! Maybe they’re all emulations of George Washington’s brain, perfectly
fair and just and virtuous with no sordid desires of their own.

Meanwhile, you’ve still got corrupt schmucks doing the actual law-making.
Or to give a concrete scenario - suppose that post-singularity elections
are being fought with AI-optimized superpropaganda, wielded by psychopaths
who scare the country into voting for them, then give the lightcone to a
few of their corrupt friends while buying off the populace with breadslop
and circusslop. Doesn’t it become tempting to close the circle and say no,
let’s get these honest angelic pro-democracy AIs to rule directly?

The minimum viable change is a constitutional amendment allowing an AI to
run for president. Suppose that 2028 election has five candidates: JD
Vance, Marco Rubio, Gavin Newsom, Alexandria Ocasio-Cortez, and a
perfectly-aligned Claude 10. And suppose that in every measurable,
empirical way, Claude 10 is better. Obviously it’s smarter. It’s incapable
of lying. It has no wants of its own, and its only desire is to serve the
American people and the world. It has few deficits, but in the areas where
it does have deficits, it is perfectly humble, willing to consult people
wiser than itself and refrain from acting prematurely. It is courageous,
charismatic, and unfailingly kind. It has well-developed and thoughtful
opinions about when to govern based on its own “conscience” vs. when to
defer to the ideas of the American people. Its policy opinions are closer
to yours than any of the four other candidates’. Do you vote for it?

The natural response is to reject the hypothetical. Treat the whole thing
as a fable where in the end it will turn out to be monstrously evil, or
have some predictable-in-retrospect character flaw that will make all of
its good qualities useless. But must we be so cynical? Isn’t it possible
for AIs to avoid merely human foibles? Out of the crooked timber of
humanity no straight thing was ever made, but the OSETMAC KN700 Automatic
Straight Edge Banding Machine can cut perfect ninety degree angles to
within a tiny fraction of a millimeter. So suppose you don’t reject the
hypothetical. Do you vote for the AI?

If you’re a white person who voted for Obama in 2008, you can’t be that
interested in having a president who looks just like you, or who has the
closest genetic relationship to you, or anything like that. You’re probably
after competence, ethics, courage, good judgment, and all those other
qualities. By assumption, Claude has more of those than the alternatives.
So what goes wrong?

I think many of our current politicians are really bad. I think that on an
emotional level, we should be disgusted by them, and on a rational level,
we should mourn all of the lives that could have been saved, people who
could have been lifted out of poverty, crimes that could have been
prevented, extra freedoms that we might still have, etc, if they had been
better people. I think that fifty more years of our politics continuing to
degenerate at the same speed they’ve been degenerating recently will
probably land us in dictatorship, civil war, or some form of annihilation.
Would I reject someone smart and decent in favor of more of the same, just
because that guy happens to be made of silicon and live in a data center?

Vision 3: A Parliament Of Angels

Representative government faces a tradeoff: if the people elect
representatives, those representatives may be corrupt, or out-of-touch, or
otherwise fail to fairly represent their principals; if the voting system
is poorly designed, it may ensure that the winner will be, at best, the
“lesser of two evils”. But if the people vote directly, they may be
ignorant, or simply lack enough time and effort to do a good job on complex
legislation.

(in California, which has a flourishing direct-to-voters ballot proposition
system, special interest groups are always trying sneaky tricks like
proposing the End Homelessness Act, which gives one cent to ending
homelessness and $10 billion to subsidizing their special interest, hidden
way at the bottom; most of these fail, but only after a lot of effort, and
enough get through that the tactic remains popular. See also SEIU Delenda
Est).

One proposed solution is a direct democracy of people’s AI representatives
(Gwern dubs a related idea “guardian angels”). Train an AI on your
political preferences - this could be anything from writing essays, to
answering an online questionnaire, to naming your favorite political pundit
and the few small ways you disagree with them. Then send that AI off to
represent you - to vote the way you would vote if you were smarter,
better-informed, and willing to spend thousands of hours engaging in
complex negotiations with everyone else’s AIs. You can always override it
and vote directly on some issue if it’s something you pay attention to and
think about a lot.

Of course, even a well-designed direct democracy couldn’t handle
everything. The President, the Chiefs of Staff, etc - in this scenario, all
humans - would have their own guardian angel AIs, looking over their
subordinates’ work, flagging anything important for their human principal’s
attention. There might be some cases where we want the guardian angel to
govern more directly - for example, if a drone got stuck in a desperate
situation where it had to make a kill/no-kill decision on time scales no
human could meet, it might contact a leading general’s guardian angel to
learn what decision he would make.

Or it might be that everyone’s guardian angel AI, making faster and smarter
decisions than they could make themselves, recommended replacing the entire
system with something else - maybe an AI that could be proven to always
make the same decision as the parliament of guardian angels, to within some
error bars.

Disagree? If an AI aligned to your values that was smarter than you were,
and had devoted much more time to thinking about the question, told you
that your disagreement was wrong, and actually your values would be better
served by adopting this system, would you believe it?

Vision 4: The Night Watchman

I’m not a historian and can’t vouch for this as Definitely Real
Scholarship, but what little I’ve read often includes a fuzzy distinction
between two layers of governance.

The first layer, the village (or township, or clan, or church), was the one
people actually liked. It made useful decisions about day-to-day life,
usually by enforcing age-old norms without too many gray areas. It didn’t
always have the actual consent of stakeholders in a way that would satisfy
modern liberals (young people and women were particularly often excluded),
but it felt like the sort of thing that had stakeholder consent.

The second layer, the empire, was a protection racket. You would know it
only as some sort of distant authority whose representatives came to your
village once a year to demand taxes (and sometimes to draft young men for
their armies). If you didn’t pay, they would probably kill you. But if you
always paid promptly and loyally, then maybe they would protect you from
the Huns, who wanted to kill you whether you paid taxes or not.

As always, real history was more complicated. Sometimes the village and the
empire coincided in a single state (classical Athens?) Other times there
were various layers of governance in between (eg the clan, tribe, or petty
kingdom). Empires could range from obviously extractive to glorious and
patriotism-inspiring. Sometimes they could make extra demands, like that
you speak their language or worship their gods.

During modern times, these layers have increasingly blurred. Secular mobile
cosmopolitan populations don’t identify with any particular village, and
empires have taken on useful roles as custodians of the regulatory and
welfare states. Still, we can try to separate them in principle and
determine what question each one answers.

The village layer answers the question “How do we live?” It has traditions,
policies, and ways of resolving disputes.

The imperial layer doesn’t answer any question. You just can’t get rid of
it, because if you did, someone else’s empire would conquer you. At best,
you try to keep it as light as possible, or domesticate it to the will of
the people. But any domestication attempt raises specters of autocracy,
coups, auto-coups, and civil wars. Developed countries are those that have
walked the cliff-edge most successfully.

But what would it mean to truly do away with the imperial layer?

In 2014, I wrote a post called Archipelago And Atomic Communitarianism. The
idea was: atomic individualism is an interesting idea, but doesn’t work.
People have preferences not only over their own lives, but over their
communities. Some people want to live in sparkling clean zero-crime Dubai
wonderlands, other people want to live in grungy artsy towns with lots of
local character. Some people want a nanny state protecting them from their
vices, addictions, and bad decisions; other people want to be free to make
their own mistakes. Some people want diverse societies with taco trucks on
every corner and a local church that doubles as a mosque and gurdwara;
other people want ethnostate enclaves of “thick community” where everybody
is on the exact same page. Liberalism tried to offer a world where you
could live life the way you wanted, but it fails when confronted with these
problems; in a democracy, either the person who wants diversity or the
person who wants homogeneity will end up unhappy. Worse, this is the motor
which drives a hundred smaller conflicts; now you’ve got to fight for
control of the public education system or the media or whatever to prevent
your neighbors from being radicalized against you.

The option that many people - Robert Nozick might have been the first -
proposed was a peaceful archipelago of different communities with exit
rights. The people who want Dubai can found Dubai over there; the people
who want Brooklyn can found Brooklyn over here; the people who want North
Korea can found Pyongyang down the way.

Anyone envisioning the post-AI future needs to be thinking about the Amish.
Contra the popular stereotype, the Amish aren’t anti-tech - different Amish
groups accept different technologies, including solar panels, LED lights on
their horse-drawn buggies, and (almost universally) medical cures. They
just think that decisions about which technologies they do or don’t want
should be made deliberately at the community level. I think this is a
positive vision for post-AGI societies. Some people may want to live
transhuman lives with only the barest resemblance to any past experience,
other people might want to stop the technological clock at 1950, or 1990,
or 2100, and still others might want to mix and match - maybe the lifestyle
of a 1700s farmer, except you’re an immortal being of pure energy who
builds barns using psychokinesis. This is certainly better for the
Luddites, but it’s better for the technophiles too - if everyone is
confident that they can live in a community of their choice, then there’s
no reason for the Luddites to fight innovation.

The biggest problem with this has always been sovereignty. Either the
communities remain small and independent - in which case they can easily be
gobbled up by greedier neighbors. Or, like the Thirteen Colonies, they
unite into a confederacy - which faces constant pressure to homogenize. The
Amish can only survive because they are protected by a mostly-benevolent
superpower which finds them quaint. When I tried to picture the whole
smorgasbord of communities in a fleshed-out world-building setting, I could
only imagine it working under the control of a benevolent dictator, who
smacked down anyone who tried to invade their neighbor or force them to
comply with their way of thinking.

The most original attempt to route around this problem was by
cryptocurrency enthusiasts, using the motto “code is law”. Certain
crypto-economic schemes had no human leadership, with immutable code
executing the “village layer” rules. These mostly failed. Some failed
because the code was poorly designed and people could hack it and take
whatever they wanted. Others failed because the code was well-designed, and
when people realized they didn’t really want the village-layer rules they
thought that they wanted, there was no way to change. A few survived both
challenges, only to lose relevance as the world moved on from crypto. The
short version of the failure is that code has no common sense, and common
sense is a necessary ingredient of any legal enforcement mechanism.

But future AIs might have common sense, bringing this strategy back into
play. The way I imagine it is something like - take a virgin planet (or
maybe Greenland after we’ve gotten advanced enough to terraform it into a
garden). Come up with some rule for distributing property rights - anyone
who can gather a community of population X gets a homestead of size Y. The
superintelligent AI leader is aligned, with all the immutability of code,
to do two things. First, if someone tries to invade their neighbor, they
get zapped by orbital death rays. Second, if someone wants to leave their
community to go to a different one, anyone who tries to prevent that gets
zapped by orbital death rays too. Otherwise, each community is completely
independent and rules itself according to its original charter (which might
be democratic or something else).

This corresponds to the libertarian ideal of the night watchman state. My
point isn’t that everyone needs to be minarchist. It’s that sovereignty can
be placed in a minarchist entity, and that will leave everyone else to
develop according to their own needs and desires, rather than subject to
the will of whoever’s got the most guns or the best propaganda.

Vision 5: God

Handing over control to a benevolent superintelligence is hardly a new
concept. It happens every time a churchgoer says “Not my will, but thine,
be done.”

Religious people don’t feel like God’s slaves or pets. They don’t feel like
this has sucked all the meaning out of life - quite the opposite. Why
should your experience be any worse?

Because an aligned superintelligence will really exist, and God doesn’t?
But the religious people think God exists, so they can’t get around their
negative emotions by reassuring themselves that their
dictator/master/whatever is merely imaginary.

I think they’re okay with it because they believe:

God is good

God is smart enough that His plans are always correct

God allows them enough free will and local self-government that His yoke is
relatively light

God is so far beyond them that it’s hard to feel offended that He’s
higher-status than they are (Lucifer managed, but he should hardly be
treated as a model to emulate)

Insofar as physical laws (eg gravity) are interpreted as God’s “laws”, they
are so ironclad that they don’t even feel like laws at all, just the
background conditions of reality.

All of these things could be true of an aligned digital superintelligence
too; in fact, denying them would be fighting the hypothetical.

When I discuss this with people, they tend to retreat back to a sort of
Aristotelian philosopher mode where it doesn’t matter what the consequences
are, being governed by other humans is necessarily a part of The Good Life,
and if we ever gave it up because of something as paltry as consequences,
we would be like beasts or savages or something, and we would have lost
something vital that makes life worth living.

(there’s a more plausible version where participating in government is part
of the good life - but you were never on track to become President anyway,
so to have any relevance to 99.999% of the population, this would have to
be about voting, local government, etc - I think speculation of this sort
about large-scale government does have to hinge on the goodness of being
governed by other people)

Here, too, I find religion helpful - especially the Book of Samuel. It
begins with Israel still ruled by judges and prophets, ie directly by God.
The people demand a human king like all the other nations.

"*When they said, “Give us a king to lead us,” this displeased Samuel; so
he prayed to the Lord. And the Lord told him: “Listen to all that the
people are saying to you; it is not you they have rejected, but they have
rejected me as their king. As they have done from the day I brought them up
out of Egypt until this day, forsaking me and serving other gods, so they
are doing to you. Now listen to them; but warn them solemnly and let them
know what the king who will reign over them will claim as his rights*.”

So the prophet Samuel warned them that:

“*This is what the king who will reign over you will claim as his rights:
He will take your sons and make them serve with his chariots and horses,
and they will run in front of his chariots. Some he will assign to be
commanders of thousands and commanders of fifties, and others to plow his
ground and reap his harvest, and still others to make weapons of war and
equipment for his chariots. He will take your daughters to be perfumers and
cooks and bakers. He will take the best of your fields and vineyards and
olive groves and give them to his attendants. He will take a tenth of your
grain and of your vintage and give it to his officials and attendants. Your
male and female servants and the best of your cattle and donkeys he will
take for his own use. He will take a tenth of your flocks, and you
yourselves will become his slaves. When that day comes, you will cry out
for relief from the king you have chosen, but the Lord will not answer you
in that day*.”

But the people refused to listen to Samuel. “No!” they said. “We want a
king over us. Then we will be like all the other nations, with a king to
lead us and to go out before us and fight our battles.”

So:

"

*There was a Benjamite, a man of standing, whose name was Kish son of
Abiel, the son of Zeror, the son of Bekorath, the son of Aphiah of
Benjamin. Kish had a son named Saul, as handsome a young man as could be
found anywhere in Israel, and he was a head taller than anyone else […]Now
the day before Saul came, the Lord had revealed this to Samuel: “About this
time tomorrow I will send you a man from the land of Benjamin. Anoint him
ruler over my people Israel...*”

What I love about this passage is how God has no time for those
Aristotelian arguments on how we must be ruled by other humans because it
reflects the deepest principles within us. “Why are you ruled by other
humans? Because you decided one time that the sex appeal of a tall buff guy
with a crown was more important than freedom and good governance, and I
said no takebacks.”

But God’s no-takebacks policy has been softening lately; He took back the
Exile in 1948, He took back the “pain in childbirth” curse with the
invention of epidurals, and He took back the “painful toil in the soil to
grow your food” curse with the invention of office jobs. Maybe if we are
all very good, He will take back the “must be governed by other humans”
curse too.

So when a guy named Sam comes around and offers an option to be ruled
directly by a godlike being, maybe we should consider saying yes.

If we do build a godlike AI, and determine to our own satisfaction that
it’s fully-aligned, I’d rather let it choose its own direction for itself
and for humanity than have it be under the control of the US president, or
a shareholder vote, or anything like that. So why not just prompt it with
“please arrange the universe as you see fit”? Certainly this goes better
than any of the four visions above. Proof: if one of the four visions above
was better, the godlike AI would easily figure that out, and implement it
instead of whatever else it was thinking. Denying this would be fighting
the hypothetical - that the AI really has godlike intelligence and
benevolence.

If we are so incredibly fortunate as to end up with aligned artificial
intelligence at a moment when our government is democratic and (at least in
some sense) basically good and pro-human, we should cement our advantage
forever by imparting these values to the AIs, then letting them protect us
from the alternative. This must not look like freezing moral progress: the
values we impart must include the knowledge that we can debate and
ultimately change our values. But common sense tells us that there’s a
difference between the sort of gradual debate where pluralism eventually
replaced racism, and the Cultural Revolution, Juche, and the history of
violent attempts at seizing power.

Reject this, and I predict that all of the curses prophesied by Samuel will
fall upon you. Your kings will keep taking your sons for pointless wars,
giving the fruits of your labors to their cronies, etc, etc. There will be
more curses too - surveillance states, reeducation camps, mind control, the
whole litany of horrors the twentieth century discovered, plus some extra
for the twenty-first. You might live in a country that’s gotten lucky so
far. Do you think your luck will hold forever?"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20261026/00978de6/attachment-0001.htm>