[extropy-chat] Fools building AIs (was: Tyranny in place)

[Eliezer S. Yudkowsky]

Rafal Smigrodzki wrote:
> On 10/4/06, Eliezer S. Yudkowsky <sentience at pobox.com> wrote:
> 
>>No one gives a damn about AI research, and until that changes, changes
>>in other government policies aren't going to affect anything one way or
>>the other.
> 
> ### You are right here, but you may dismiss my worries a bit too
> early: One day in the not-too-distant future some high-level
> government officers will approve major funding for a general AI, and
> today's changes in political culture are likely to have an impact on
> the moral and intellectual qualities of these officers.

It really doesn't matter whether an improperly shaped intelligence 
explosion is set off by altruistic idealists, or patriots, or 
bureaucrats, or terrorists.  A galaxy turned into paperclips is just as 
much of a tragedy either way.

> As you note,
> even the best intentions of idealistic researchers may backfire
> horribly, but the intellectual offspring of evil functionaries is
> simply guaranteed to fry us all.

Not "may" backfire horribly.  Rafal, you work in biology.  You know from 
experience that biology is difficult.  You've probably also met people 
who think that biology is easy enough for them to make up cool-sounding 
theories about it.  Think about what behavior you've witnessed from most 
AGI wannabes.  Ask yourself if they seem to understand intelligence as 
solidly as you understand biology.  Now imagine someone with that degree 
of understanding of biology, trying to build something really wonderful 
and exotic - say, a retrovirus that reverses Alzheimer's disease.  It 
doesn't matter if they're motivated by the will to heal, or pure greed - 
they can't do it at that level of understanding, end of story.

If some high-level government officers approve major funding for a 
general AI, they'll find a prestigious AI researcher, someone who's been 
failing at the problem longer than anyone else, and appoint a 
blue-ribbon committee containing not a single researcher under the age 
of 40 to oversee the project.

Google might be worrisome someday.  But for now, Google's founders 
apparently believe that it is Google's destiny to become an AI and this 
will happen by magic, when their databases grow large enough to be 
blessed by the Emergence Fairy.  I am more worried about a single one of 
Google's resident geniuses spending their days off on an AI project. 
Partial understanding is dangerous; great big computer clusters are not. 
  Which would you be more afraid of, Rafal, an environmental extremist 
with a grade school education and a multibillion dollar state-of-the-art 
medical research lab, or an environmental extremist with a Ph.D. and a 
hundred thousand dollars?

Newton wasted much of his life on Christian mysticism, so it's possible 
to make great discoveries and still not understand how to think.  But 
the art of an FAI creator is thought itself - to see cognition as an 
engine which accomplishes work.  I cannot visualize someone discovering 
the basic organizing principles of intelligence, in a scientific field 
presently full of confusion and dismay, without their having an 
*intuitive* appreciation of how rationality works.  It would be like 
Newton not being able to see mathematics incarnate in a falling apple. 
It would be like Carnot not being able to visualize heat flows.  Maybe 
someday there'll be textbooks that teach idiots how to build AI, but to 
discover it yourself, you need an *intuitive* appreciation of the 
results achieved by cognitive workflows.  You cannot master that art 
*from scratch* and afterward still be so poor a rationalist as Newton. 
No Sharia zombie could get one tenth of the way to independently 
discovering how to build and shape an AGI, and still remain a Sharia zombie.

A genius fool might stumble over powerful forces they can't control.  A 
genius fool might build something that sorta works, in a poorly 
understood way, but works just well enough to cross the threshold of 
recursive self-improvement.  A genius fool might spend their days 
working with evolutionary algorithms, knowing nothing of Bayescraft. 
After all, natural selection originally built humans using no abstract 
understanding whatsoever.  So, yes, genius fools are dangerous - but not 
because they might decide to shape a Sharia enforcer.  Genius fools are 
dangerous because they can't shape minds at all.  It doesn't matter 
whether their intentions are good or bad, because the outcome bears no 
relation to their intentions for it.

It requires a precise understanding to master the shaping art, to set 
off an intelligence explosion that relates *at all* to your original 
intentions.  If there is any recognizable resemblance you must have 
known a very great deal indeed.  It requires considerably *less* 
understanding than that, to appreciate how powerful the forces are which 
you intend to mess with.  No one can know how to build a nuclear reactor 
that operates in a state of controlled criticality, without being able 
to calculate the doubling time of a chain reaction.  No one can 
understand intelligence and still think of an intelligence explosion as 
a weapon in short-term political squabbles.  An astrophysicist knows the 
power output of a star, and that a star is brighter than a campfire.  An 
astrophysicist is not going to say, "Hey, let's set off a supernova to 
bust in that terrorist bunker."  Maybe the astrophysicist's superiors 
fail to comprehend anything about a supernova except that it makes a 
bang - but the astrophysicist can't help but know what would happen.  If 
the astrophysicist is ordered to do it anyway, told to set off a 
supernova or face a court martial, the astrophysicist is not going to 
shrug and go ahead with it.  Imagine this as a *literal* scenario, not a 
metaphor.  If you want to keep your job, you might try all sorts of 
dodges, but you wouldn't actually set off a supernova - not when you 
knew damned well that the Earth would end up as not-even-vapor.

So can we have enough of this silly scenario where someone creates *and 
successfully shapes* an intelligence explosion, while simultaneously not 
noticing that they are messing with powers vast enough to reconfigure 
galaxies?  Can we stop pretending that someone might build *and shape* 
an AGI, by the exercise of their precise understanding, for the sake of 
a cute little toy weapons system?  A genius fool might accidentally fry 
the planet, turn our galaxy into paperclips.  But no genius fool could 
accomplish any specific purpose, anything with a nonzero correlation to 
what they originally had in mind.  Paperclips are paperclips - whether 
that outcome is brought about by terrorists or patriots, universities or 
Google, idealists or villains.  All you're arguing about is whose logo 
will look the prettiest on the tombstone.

This is a counterintuitive point, I know.

It is much more satisfying to cheer the Blues, or boo the Greens, or 
whatever your accustomed chariot-racing allegiance may be.

But this problem is more difficult than that, and old habits will not 
solve it.

-- 
Eliezer S. Yudkowsky                          http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence