[extropy-chat] Fools building AIs (was: Tyranny in place)
Eliezer S. Yudkowsky
sentience at pobox.com
Wed Oct 4 21:04:04 UTC 2006
Rafal Smigrodzki wrote:
> On 10/4/06, Eliezer S. Yudkowsky <sentience at pobox.com> wrote:
>>No one gives a damn about AI research, and until that changes, changes
>>in other government policies aren't going to affect anything one way or
> ### You are right here, but you may dismiss my worries a bit too
> early: One day in the not-too-distant future some high-level
> government officers will approve major funding for a general AI, and
> today's changes in political culture are likely to have an impact on
> the moral and intellectual qualities of these officers.
It really doesn't matter whether an improperly shaped intelligence
explosion is set off by altruistic idealists, or patriots, or
bureaucrats, or terrorists. A galaxy turned into paperclips is just as
much of a tragedy either way.
> As you note,
> even the best intentions of idealistic researchers may backfire
> horribly, but the intellectual offspring of evil functionaries is
> simply guaranteed to fry us all.
Not "may" backfire horribly. Rafal, you work in biology. You know from
experience that biology is difficult. You've probably also met people
who think that biology is easy enough for them to make up cool-sounding
theories about it. Think about what behavior you've witnessed from most
AGI wannabes. Ask yourself if they seem to understand intelligence as
solidly as you understand biology. Now imagine someone with that degree
of understanding of biology, trying to build something really wonderful
and exotic - say, a retrovirus that reverses Alzheimer's disease. It
doesn't matter if they're motivated by the will to heal, or pure greed -
they can't do it at that level of understanding, end of story.
If some high-level government officers approve major funding for a
general AI, they'll find a prestigious AI researcher, someone who's been
failing at the problem longer than anyone else, and appoint a
blue-ribbon committee containing not a single researcher under the age
of 40 to oversee the project.
Google might be worrisome someday. But for now, Google's founders
apparently believe that it is Google's destiny to become an AI and this
will happen by magic, when their databases grow large enough to be
blessed by the Emergence Fairy. I am more worried about a single one of
Google's resident geniuses spending their days off on an AI project.
Partial understanding is dangerous; great big computer clusters are not.
Which would you be more afraid of, Rafal, an environmental extremist
with a grade school education and a multibillion dollar state-of-the-art
medical research lab, or an environmental extremist with a Ph.D. and a
hundred thousand dollars?
Newton wasted much of his life on Christian mysticism, so it's possible
to make great discoveries and still not understand how to think. But
the art of an FAI creator is thought itself - to see cognition as an
engine which accomplishes work. I cannot visualize someone discovering
the basic organizing principles of intelligence, in a scientific field
presently full of confusion and dismay, without their having an
*intuitive* appreciation of how rationality works. It would be like
Newton not being able to see mathematics incarnate in a falling apple.
It would be like Carnot not being able to visualize heat flows. Maybe
someday there'll be textbooks that teach idiots how to build AI, but to
discover it yourself, you need an *intuitive* appreciation of the
results achieved by cognitive workflows. You cannot master that art
*from scratch* and afterward still be so poor a rationalist as Newton.
No Sharia zombie could get one tenth of the way to independently
discovering how to build and shape an AGI, and still remain a Sharia zombie.
A genius fool might stumble over powerful forces they can't control. A
genius fool might build something that sorta works, in a poorly
understood way, but works just well enough to cross the threshold of
recursive self-improvement. A genius fool might spend their days
working with evolutionary algorithms, knowing nothing of Bayescraft.
After all, natural selection originally built humans using no abstract
understanding whatsoever. So, yes, genius fools are dangerous - but not
because they might decide to shape a Sharia enforcer. Genius fools are
dangerous because they can't shape minds at all. It doesn't matter
whether their intentions are good or bad, because the outcome bears no
relation to their intentions for it.
It requires a precise understanding to master the shaping art, to set
off an intelligence explosion that relates *at all* to your original
intentions. If there is any recognizable resemblance you must have
known a very great deal indeed. It requires considerably *less*
understanding than that, to appreciate how powerful the forces are which
you intend to mess with. No one can know how to build a nuclear reactor
that operates in a state of controlled criticality, without being able
to calculate the doubling time of a chain reaction. No one can
understand intelligence and still think of an intelligence explosion as
a weapon in short-term political squabbles. An astrophysicist knows the
power output of a star, and that a star is brighter than a campfire. An
astrophysicist is not going to say, "Hey, let's set off a supernova to
bust in that terrorist bunker." Maybe the astrophysicist's superiors
fail to comprehend anything about a supernova except that it makes a
bang - but the astrophysicist can't help but know what would happen. If
the astrophysicist is ordered to do it anyway, told to set off a
supernova or face a court martial, the astrophysicist is not going to
shrug and go ahead with it. Imagine this as a *literal* scenario, not a
metaphor. If you want to keep your job, you might try all sorts of
dodges, but you wouldn't actually set off a supernova - not when you
knew damned well that the Earth would end up as not-even-vapor.
So can we have enough of this silly scenario where someone creates *and
successfully shapes* an intelligence explosion, while simultaneously not
noticing that they are messing with powers vast enough to reconfigure
galaxies? Can we stop pretending that someone might build *and shape*
an AGI, by the exercise of their precise understanding, for the sake of
a cute little toy weapons system? A genius fool might accidentally fry
the planet, turn our galaxy into paperclips. But no genius fool could
accomplish any specific purpose, anything with a nonzero correlation to
what they originally had in mind. Paperclips are paperclips - whether
that outcome is brought about by terrorists or patriots, universities or
Google, idealists or villains. All you're arguing about is whose logo
will look the prettiest on the tombstone.
This is a counterintuitive point, I know.
It is much more satisfying to cheer the Blues, or boo the Greens, or
whatever your accustomed chariot-racing allegiance may be.
But this problem is more difficult than that, and old habits will not
Eliezer S. Yudkowsky http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence
More information about the extropy-chat