[extropy-chat] The emergence of AI

[Russell Wallace]

On Sat, 04 Dec 2004 22:50:21 -0500, Robin Hanson <rhanson at gmu.edu> wrote:
> Well I grant that some things have stayed constant for long periods of
> economic growth, and that it is important to be able to analyze growth
> in some things while holding other things static.  But in the future
> the human genome, material composition of the Earth and energy input
> from the sun, and even the laws of physics may well change.  If anything
> is going to change these things, it will be further economic growth.

(Nitpick: We can't change the ultimate laws of physics, more or less
by definition; if we do find a way to change things like the mass of
the proton, then that'll be in accordance with some deeper laws, which
are what I'll refer to as the laws of physics here.)

But yes, lots of things that have stayed static in the past might
change in the future. The laws of physics will stay constant, of
course. If those are the only static part, what will they drive the
dynamic part towards? Optimal self-replication; in practical terms,
what we'll likely get will be an expanding sphere of nonsentient von
Neumann probes, converting all matter into copies of themselves...
i.e. just as sure and total a loss as if we simply blew up the planet
tomorrow.

So in a sense the challenge of the Singularity, and of Friendly AI, is
to find a way to expand the static part; to keep those things we value
static, while driving enough dynamic change to not get left behind.

- Russell