[ExI] Critical take on The Age of Em

Robin D Hanson rhanson at gmu.edu
Wed Jun 22 13:03:43 UTC 2016


On Jun 21, 2016, at 9:36 PM, Rafal Smigrodzki <rafal.smigrodzki at gmail.com<mailto:rafal.smigrodzki at gmail.com>> wrote:
On Tue, Jun 7, 2016 at 9:13 PM, Robin D Hanson <rhanson at gmu.edu<mailto:rhanson at gmu.edu>> wrote:

I suppose I count as having read it, but not as an independent evaluator. :)

### Finally I read Age of Em, and quite a few of its reviews as well. Let me go meta on them and commiserate with you: Most of the reviews are completely missing the point. They tend to apply a literary standard ("How do I feel about it") to a work meant more like a documentary, which should be judged by "How likely is it to be correct". Scott Alexander's review is most apt, and it is hard to add something substantive to any subject once Scott is done with it. But I'll try.

Well with Oxford as publisher I am trying to straddle the line between an academic and a trade book. I would, as you indicate, rather be judged on “is it correct”, but trade books are more judged by “how do you feel about it”.

Obviously, the largest objection is of low likelihood of the em future. I know, you cleverly defuse this criticism by agreeing that the likelihood of this scenario is 1/1000 and you still proceed with analysing it. But didn't you say that in this book you aim to straightforwardly apply the methods and knowledge of today's science to an aspect of the future? Yet when 100% of experts on AI say em is not likely to happen before AI, you do not take their opinion at face value. Something does not fit here.

100% does not remotely sound correct. I request a citation for that figure.

I think that the odds in favor of em as described before AI are even lower than 1/1000. Brains are difficult. Chips are easier. For em before AI you need it all - massive computing power, *and* a very detailed knowledge of brains and cognition in general, *and* a fantastically detailed scan of a particular human.

Recently, a local intelligence explosion occurred at Google - a program taught itself to play Space Invaders, then Go, just by massively thinking about it (millions of rounds of reinforced learning from playing against itself). Intelligence explosions are a reality, and they already produce AI without recourse to human-derived structure, given enough computing power and enough learning examples. So for em before AI you would expect massive computing power, successful reinforced deep learning and billions of dollars to go nowhere for many decades. That's the first hurdle, and it's a huge one.

There has indeed been a welcome burst of progress in machine learning lately, but we have seen bursts of progress in AI before, and we are still lightyears away from achieving human level AI. At the average rates of progress we’ve seen in the past we have centuries still to go. Also you aren’t using the phrase “intelligence explosion” the way it is usually used. But the usual definition, these observed bursts of progress don’t count.

But let's assume that AI will be stuck on Go for a 100 years. Neuroscience will catch up, and we will have the detailed knowledge of human brain function, necessary and sufficient to devise an emulation. Obviously, this won't happen in one fell swoop. There will be dozens or hundreds of partial emulations, painstakingly building up various aspects of a brain from thousands of data sources. At some point you will be able to connect multiple such partial networks into a generic model of a human mind. It will not have individual memories but it will have the ability to accept appropriately formatted input data, self-organize like a human brain and perform signal processing like a human brain, generating individual memories in this process, just like a human brain does. For your em scenario, this generic human model would have be stuck in neutral while a detailed individual scan is made and the knowledge derived from the generic human is used to transform this scan into a working individual emulation. It is very unlikely that generic mind models fail to materialize before the individual em you analyze.

Imagine someone gave you dozens of examples of financial spaghetti code from decades ago, all of which do a similar range of financial tasks. And all you have is the object code, not the source code. It would be very hard to abstract from those examples a “generic” financial system capable of doing those financial tasks. That isn’t a remotely easy task. To create a generic brain you have to abstract usefully from the spaghetti object code that is the human brain.

But, let's assume that we have individual em before AI and before generic em. In this situation I come to my second major objection: Your pervasive assumption that em will remain largely static in their overall structure and function. I think this assumption is at least as unlikely as the em-before-AI assumption. Imagine you find yourself in a world to which you are not well adapted at all, as a being evolved in the African veldt that had to tear flesh and woo women to make copies of self, but now you have the detailed knowledge of your own mind, the tools to modify it, and the ability to generate millions of copies to try out various modifications. How long would it take you to remake yourself to fit the silicon plains you live in? I know, you do analyze this possibility, you consider some options but in the end you still assume ems will be just like us.

If you had a huge piece of legacy object spaghetti code for a financial system, you would also find it hard to usefully change that if you did not understand it. And the human mind is vastly more complex than any known legacy financial code.

Of course, if ems are not like us, then a lot of the detailed sociological research produced on humans would not be very applicable to their world and the book would have to be shorter, but then it might be a better one. In one chapter you mention that lesbian women make more money and therefore lesbian ems might make money as well. This comes at the end of many levels of suspension of disbelief, making the sociology/gender/psychology chapters quite exhausting.

Can you say which direction that you think the ability to self-modify brains will change the relative earning ability of lesbian women relative to non-lesbian woman? If not, the value for women today remains our best estimate for the future.

Now, lest anybody thinks I am trashing Age of Em, I did enjoy reading many or maybe even most chapters. I liked the chapters analyzing basic physics of em computation, reversible computation, the intersection of economics and variable speed of ems, the economics of homo electro-economicus (as the ems might turn out to be), governance by auction among the economically literate and many other ideas.

Thanks; I’m glad you enjoyed a lot. :)

I feel that em clans would be even more prominent than you say. I think you give too much of a short-shrift to open-source ems. I understand the reasons but I disagree - the local training costs are likely to be low for the vast majority of generic workplaces and teaching open-source ems will be better than hiring proprietary ones, for security and other reasons.

There can be many other fixed costs in labor markets besides training, and all fixed costs discourage open source ems.

I don't think that ems will have the arc of life you describe (formation, training, work, retirement) - it will be possible to restructure aging ems to make them mentally young enough again, which goes directly against the pervasive assumption of (almost) static humanity you make.

I guess you don’t believe in software rot? You think it possible to teach real legacy software systems and make them young again? People try to do this with refactoring, but it is very hard and has only limited success.

Skipping the detailed sociology chapters would do the book good, and instead you could have explored the alternative scenarios you briefly mention in one of the last chapters. Why not try to look at the interactions between AI (a form of capital) and uploaded ems (a mixture of capitalists and laborers) in an AI-before-em scenario?

That will have to wait for another book.

But overall, the book is a nerdvana for the transhumanist and can be recommended for all who like to think (as opposed to merely feel) about the future.

Yay! :)

Robin Hanson rhanson at gmu.edu<mailto:rhanson at gmu.edu>
Future of Humanity Inst., Oxford University
Assoc. Prof. Economics, George Mason University
See my new book: http://ageofem.com









-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20160622/74ddb9b6/attachment.html>


More information about the extropy-chat mailing list