[ExI] Critical take on The Age of Em

[Rafal Smigrodzki]

On Tue, Jun 7, 2016 at 9:13 PM, Robin D Hanson <rhanson at gmu.edu> wrote:

I suppose I count as having read it, but not as an independent evaluator. :)
>
> ### Finally I read Age of Em, and quite a few of its reviews as well. Let
me go meta on them and commiserate with you: Most of the reviews are
completely missing the point. They tend to apply a literary standard ("How
do I feel about it") to a work meant more like a documentary, which should
be judged by "How likely is it to be correct". Scott Alexander's review is
most apt, and it is hard to add something substantive to any subject once
Scott is done with it. But I'll try.

Obviously, the largest objection is of low likelihood of the em future. I
know, you cleverly defuse this criticism by agreeing that the likelihood of
this scenario is 1/1000 and you still proceed with analysing it. But didn't
you say that in this book you aim to straightforwardly apply the methods
and knowledge of today's science to an aspect of the future? Yet when 100%
of experts on AI say em is not likely to happen before AI, you do not take
their opinion at face value. Something does not fit here.

I think that the odds in favor of em as described before AI are even lower
than 1/1000. Brains are difficult. Chips are easier. For em before AI you
need it all - massive computing power, *and* a very detailed knowledge of
brains and cognition in general, *and* a fantastically detailed scan of a
particular human.

Recently, a local intelligence explosion occurred at Google - a program
taught itself to play Space Invaders, then Go, just by massively thinking
about it (millions of rounds of reinforced learning from playing against
itself). Intelligence explosions are a reality, and they already produce AI
without recourse to human-derived structure, given enough computing power
and enough learning examples. So for em before AI you would expect massive
computing power, successful reinforced deep learning and billions of
dollars to go nowhere for many decades. That's the first hurdle, and it's a
huge one.

But let's assume that AI will be stuck on Go for a 100 years. Neuroscience
will catch up, and we will have the detailed knowledge of human brain
function, necessary and sufficient to devise an emulation. Obviously, this
won't happen in one fell swoop. There will be dozens or hundreds of partial
emulations, painstakingly building up various aspects of a brain from
thousands of data sources. At some point you will be able to connect
multiple such partial networks into a generic model of a human mind. It
will not have individual memories but it will have the ability to accept
appropriately formatted input data, self-organize like a human brain and
perform signal processing like a human brain, generating individual
memories in this process, just like a human brain does. For your em
scenario, this generic human model would have be stuck in neutral while a
detailed individual scan is made and the knowledge derived from the generic
human is used to transform this scan into a working individual emulation.
It is very unlikely that generic mind models fail to materialize before the
individual em you analyze.

But, let's assume that we have individual em before AI and before generic
em. In this situation I come to my second major objection: Your pervasive
assumption that em will remain largely static in their overall structure
and function. I think this assumption is at least as unlikely as the
em-before-AI assumption. Imagine you find yourself in a world to which you
are not well adapted at all, as a being evolved in the African veldt that
had to tear flesh and woo women to make copies of self, but now you have
the detailed knowledge of your own mind, the tools to modify it, and the
ability to generate millions of copies to try out various modifications.
How long would it take you to remake yourself to fit the silicon plains you
live in? I know, you do analyze this possibility, you consider some options
but in the end you still assume ems will be just like us.

Of course, if ems are not like us, then a lot of the detailed sociological
research produced on humans would not be very applicable to their world and
the book would have to be shorter, but then it might be a better one. In
one chapter you mention that lesbian women make more money and therefore
lesbian ems might make money as well. This comes at the end of many levels
of suspension of disbelief, making the sociology/gender/psychology chapters
quite exhausting.

Now, lest anybody thinks I am trashing Age of Em, I did enjoy reading many
or maybe even most chapters. I liked the chapters analyzing basic physics
of em computation, reversible computation, the intersection of economics
and variable speed of ems, the economics of homo electro-economicus (as the
ems might turn out to be), governance by auction among the economically
literate and many other ideas. I feel that em clans would be even more
prominent than you say. I think you give too much of a short-shrift to
open-source ems. I understand the reasons but I disagree - the local
training costs are likely to be low for the vast majority of generic
workplaces and teaching open-source ems will be better than hiring
proprietary ones, for security and other reasons. I don't think that ems
will have the arc of life you describe (formation, training, work,
retirement) - it will be possible to restructure aging ems to make them
mentally young enough again, which goes directly against the pervasive
assumption of (almost) static humanity you make. Skipping the detailed
sociology chapters would do the book good, and instead you could have
explored the alternative scenarios you briefly mention in one of the last
chapters. Why not try to look at the interactions between AI (a form of
capital) and uploaded ems (a mixture of capitalists and laborers) in an
AI-before-em scenario?

But overall, the book is a nerdvana for the transhumanist and can be
recommended for all who like to think (as opposed to merely feel) about the
future.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20160621/ad590550/attachment.html>