<html><head></head><body><div><span title="hkeithhenson@gmail.com">Keith Henson</span><span class="detail"> <hkeithhenson@gmail.com></span> , 11/4/2014 6:01 AM:<br><blockquote class="mori" style="margin:0 0 0 .8ex;border-left:2px blue solid;padding-left:1ex;">On Thu, Apr 10, 2014 at 6:18 PM, Anders Sandberg <<a href="mailto:anders@aleph.se" class="mailto">anders@aleph.se</a>> wrote:
<br>> Any way of doing a formal analysis of it? We know human status gaming can be pretty destructive.?
<br>
<br>That's a really good question. I don't know.
<br>
<br>From what I see of destructive, zero sum, or negative sum status
<br>games, they seem to stem from poor intelligence or poor understanding
<br>of the object of the game. Presumably an AI would be smart enough to
<br>play the game well.
<br>
<br>We should discuss this, either over Skype, or the next time I get to Oxford.</blockquote></div><div><br></div>Totally!<div><br></div><div>My concern is that just like in the moral/value case most friendliness research has been thinking about, getting complex and fragile human social status concepts into the machine seems to be hard. And if we miss what we really mean by social status we might get powerful systems playing a zero-sum game with arbitrary markers that just *sound* like they are social status as we know it. In such a situation humans might be hopelessly outclassed and the desired integration of machine and human society never happens. So the issue to think about is how to make sure the concepts actually mesh. And that playing the game doesn't lead to pathologies even when you are smart: we know the moral game can get crazy for certain superintelligences. <br><div><blockquote class="mori" style="margin:0 0 0 .8ex;border-left:2px blue solid;padding-left:1ex;">
</blockquote></div><br>Anders Sandberg, Future of Humanity Institute Philosophy Faculty of Oxford University<div><blockquote class="mori" style="margin:0 0 0 .8ex;border-left:2px blue solid;padding-left:1ex;"></blockquote></div></div></body></html>