[ExI] Hard Takeoff

[spike]

On Behalf Of John Grigg

>...But the "classic scenario" ... seems very doubtful in my view...Am I
missing something here?  John


No Johnny, rather you are getting something here, something very
fundamental: the uncertainty inherent in AI research.  This has really
bothered me since about the mid-90s when I was introduced to the notion of
the singularity: the inherent uncertainty is often downplayed.

If we want to go with the plutonium analogy, we have some immediate
problems.  There were unknowns of course, but the behavior of a critical
mass of plutonium could be calculated, with slipsticks, nuclear cross
section tables, and the results of a number of lab tests.  The scientists
could model the feedback loops, estimate closely the outcome.  They could
calculate the risk of igniting the atmosphere and destroying all life on the
planet for instance.  The results of the tests at the Trinity site didn't
surprise the scientists present.  They were awed to the core of their
beings, but not surprised.

Intelligence in any substrate is far less predictable.  Put a bunch of
really smart scientists together and it becomes wildly unpredictable what
will happen.  

Consider my energetic reaction to Singularity Utopia.  He or she went on and
on about how everything would be just grand.  I now realize that she may
have been a creative singularity-phobe who is making a point, a good one, by
posing as a wild-eyed singularity-phile.  All the megalomania could have
been a kind of over-the-top satire, to point to the great danger of running
in to a danger zone with wild-eyed optimism.  Alternately she could be
exactly what she wrote, in which case she made a damn good point anyway,
although not the one she intended.

I am not advocating a Bill Joy approach of eschewing AI research, just the
opposite.  A no-singularity future is 100% lethal to every one of us, every
one of our children and their children forever.  A singularity gives us some
hope, but also much danger.  The outcome is far less predictable than
nuclear fission.  

Good luck to us.

spike