[extropy-chat] Darwinian dynamics unlikely to apply to superintelligence

[Eliezer S. Yudkowsky]

Perry E. Metzger wrote:

> Robin Lee Powell <rlpowell at digitalkingdom.org> writes:
> 
>>>Myself, though, I will make a strong prediction -- which is that
>>>the laws of physics and the rules of math don't cease to apply.
>>>That leads me to believe that evolution doesn't stop. That further
>>>leads me to believe that nature -- bloody in tooth and claw, as
>>>some have termed it -- will simply be taken to the next level. I
>>>don't fear this particularly, but it isn't consistent with the
>>>"everything is going to turn up roses" viewpoint.
>>
>>You've taken one sample set, Earth, and implied from the course of
>>evolution on Earth that it is a *law of physics* that violent
>>conflict occur.

> Evolution isn't something you can avoid. Deep down, all it says is
> "you find more of that which survives and spreads itself", which is so
> close to a tautology that it is damn hard to dispute. There is no
> moral superiority to a bacterium that minds its manners over one that
> overwhelms its competition. The universe on a deep level doesn't care
> which one you find more of. However, almost axiomatically, the second
> one is the one you'll find in every soil sample and the first will be
> rare or extinct.

The replicator dynamics, like all math equations, generally are provable
and hence what people would call "tautological" when applied to the real
world.  The question is whether the variables take on any interesting
values.  Price's Equation is a tautology, and, therefore, always true; the
question is whether it usefully applies.  You can apply it to pebbles on
the seashore, for example, and lo, the change in the mean value of the
"blue" characteristic for the pebbles will equal the covariance of the
"blue" characteristic with the proportional change (fitness) of each color
category of pebbles.  That is, it will be precisely, numerically equal, if
you do the calculation.  But the correlation will end up being slight, and
will probably change sign from generation to generation, because the
covariance is noise and not the causal result of physically perseverant
properties of the pebbles; and whatever categories you parse the pebbles
into, it will be a function of an arbitrary classification system, and not
physically copied genes.  And yet the math will still, technically, add up.

Although the math holds true tautologically for any consistent set of
variable definitions, it is not at all trivial to show that the math
*applies* to some physical system in the sense of the variable definitions
corresponding to simple physical properties, rather than being produced
arbitrarily.

And when the variable definitions in a replicator equation do correspond
to simple physical properties, there is still the question of whether one
is dealing with infinitesimal quantities that obey a replicator equation,
or large quantities; small handful of generations, or millions of
generations; whether there is enough selection pressure, over a long
enough period of time, to produce complex information of the sort we're
used to seeing in biology.

To sum up, natural selection *as we know it*, which is to say, natural
selection in any noticeable quantity, is not an automatic consequence of
physics.  It applies to butterflies, but not pebbles, even though the math
can be defined for both cases.  Even if blue pebbles survive some tiny
amount better, it doesn't mean that in 20,000 years all the pebbles on the
seashore will be intensely blue.

We are more likely to see the longest-burning stars, and if you were to
stretch the term far enough, you could insist that stars have
"generations" because the debris of a nova ends up being incorporated into
new stars, and so on, but because the "heritable" capacity is noise (even
though it can still be defined as a mathematical quantity) and the number
of generations so few, we do not see stars that are optimized to burn for
trillions of years, even though we can expect that the stars we see will
have been selected so as to exclude ones that fail to ignite or explode
immediately.

Correspondingly, we can expect that any SI we deal with will exclude the
set of SIs that immediately shut themselves down, and that whichever SI we
see will be the result of an optimization process that was capable of
self-optimization and preferred that choice.  But this does not imply that
any SI we deal with will attach a huge intrinsic utility to its own survival.

If you have an optimization system, and that optimization system behaves
something at least roughly like the expected utility equation, then,
regardless of the particulars of the utility function, it seems
straightforward to derive instrumental expected utility for the continued
operation of an optimization system similar to the one doing the
calculation, and the expected instrumental utility calculated in the
present time will increase with the expected fidelity of the utility
function.  This will hold true of a very large class of optimizers.

It follows that we have no reason to expect any SI we deal with to attach
a huge intrinsic utility to its own survival.  Why?  Because that's an
extremely specific outcome within a very large class of outcomes where the
SI doesn't shut itself down immediately.  There is, in other words, no
Bayesian evidence - no likelihood ratio - that says we are probably
looking at an SI that attaches a huge intrinsic utility to its own
survival; both hypotheses produce the same prediction for observed behavior.

Similarly, for any optimization process that can configure matter in ways
that it reckons will create instrumental utility, or fulfill intrinsic
utility, or avoid expected negative utility, with respect to any possible
aspect of its goal system, we should expect that optimization process to
optimize all available matter, since that action will be perceived as more
desirable than the alternative, assuming the entity implements some kind
of expected utility equation for ordering preferences over choices.

For an extremely large class of SIs, they will *all* choose to absorb all
nearby matter.  So there is no reason to suppose that they would need a
particular desire to reproduce.

I expect that most any optimization process including a Friendly SI, and
certainly including myself, would choose to defend itself from a hostile
optimization process - as an instrumental utility.  So there is no reason
to suppose that any SI we see must have a particular desire to engage in
combat.

And finally, there is no reason to suppose that the process whereby SIs
absorb matter, optimize matter, or in other ways do things with matter,
would create subregions with (a) large heritable changes in properties,
that (b) correlate to large differences in the rate at which these regions
spread or transform other matter, and that (c) this process will continue
over the thousands or millions of generations that would be required for
the natural selection dynamic to produce optimized functional complexity.

This last point is particularly important in understanding why replicator
dynamics are unlikely to apply to SIs.  At most, we are likely to see one
initial filter in which SIs that halt or fence themselves off in tiny
spheres are removed from the cosmic observables.  Almost any utility
function I have ever heard proposed will choose to spread across the
cosmos and transform matter into either (1) *maximally high-fidelity
copies* of the optimization control structure or (2) configurations that
fulfill intrinsic utilities.  If the optimization control structure is
copied at extremely high fidelity, there are no important heritable
differences for natural selection to act on.  If there were heritable
differences, they are not likely to covary with large differences in
reproductive fitness, insofar as all the optimization control structures
will choose equally to transform nearby matter.

Natural selection operates on the *covariance* between heritable
quantities and reproductive fitness, not the *correlation*, which means
that the *amount* of variation is relevant.  (When you calculate the
correlation you take the covariance between the two quantities and divide
by the product of the two standard deviations of each quantity.)  If
there's only a small amount of heritable variation, the covariance goes
down.  If there's only a small amount of variation in reproductive
fitness, the covariance goes down.  If you don't have thousands of
generations, the amount of genetic information generated by the iteration
of this covariance will be tiny, certainly not enough to account for
complex adaptations.

In short, under scenarios of the type I have seen discussed so far,
replicator dynamics do not apply to SIs.

Natural selection is not a binary thing that switches on or off depending
on whether anything that can be called "replication" occurs.  Selection
pressure can be quantified - and it is a surprisingly small optimization
effect by our standards; for example, if each couple has an average of 8
children then there can be *at most* 2 bits of information produced by
natural selection per generation, to be shared among all the quantities
subject to optimization.  With the scenarios extrapolated as usual, and
the variable definitions that are usually offered, the amount of selection
pressure that applies to SIs is infinitesimal, barring a possible initial
selection filter.

> So what sort of strategies does evolution favor? Quite a number of
> them, actually, but none of them can be characterized as "pacifist".

None of them can be characterized as conscientious objectors, barring
those that are physically invulnerable.  I don't see that Friendliness
requires being a conscientious objector to a generic optimization process
that's trying to eat you, and I certainly don't see why a generic
optimization process would choose to be a conscientious objector as an
instrumental utility, barring an exotic utility function.

Anyway, there's a heck of a difference between natural selection *building
a goal system from scratch*, like where humans come from, and applying a
anti-suicide filter to the set of SIs that are likely to pop up from
ancestral civilizations (mostly the result of runaway recursive
self-improvement, I expect, perhaps a Friendlyoid SI here and there if
someone in the ancestral civilization was implausibly competent).

> The struggle for resources is unlikely to end, because the amount of
> resource you can have in any finite volume remains finite.

Replicator dynamics assume a (large, frequent) death rate.  If
optimization processes compete to absorb *available* resources but hang on
permanently to all resources already absorbed, the replicator dynamics are
not iterated across thousands of generations.

> That leads
> me to assume that we'll continue to see evolution take place as life
> spreads through the cosmos.

If so, it will be because Friendly SIs (Friendliness being an exotic
utility function created by the intervention of humans) contain
environments with social scenarios to which replicator dynamics apply.
Even this is not a necessary assumption, and replicator dynamics might
apply to some social quantities and not others.  For example, if
sociolegal dynamics fix the maximum number of children at, say, two
children per couple, or one child every hundred years, and motivational
variations are heritable, then natural selection might apply as a floor
function that minimizes the number of sentient entities who want to have
*less* children than that.  But natural selection would be powerless to
optimize variables within the very large class of minds that chose to use
their full allotment.

On the other hand, *memes* might be around for a long, long time.

> That, in turn, leads me to assume that
> we'll continue to see "nature bloody in tooth and claw", although
> perhaps it will become "nature bloody in assembler and particle beam"
> or other gadgetry far beyond our understanding.

It looks to me like, whether humanity survives or fails, the era of
natural selection as we know it - "bloody in tooth and claw" - is ending.

-- 
Eliezer S. Yudkowsky                          http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence