<html><head></head><body><strong>Från:
</strong>
Giulio Prisco <giulio@gmail.com>
<strong></strong><br><br><blockquote class="mori" style="margin:0 0 0 .8ex;border-left:2px blue solid;padding-left:1ex;">Anders, the paperclip maximizer is a good example indeed. For all we<br>know, it could reason that humans are the only users of paperclips in<br>the known universe, so that humans give a cosmic meaning to paper clip<br>production, so that humans must be revered and protected... My point<br>is that we don't know - just like Columbus didn't know what he would<br>find at the end of his journey.</blockquote><br>But that is not my example. That would be a paperclip-meaning maximizer or something. I was talking about something that evaluates value as being proportional to the number of paperclips (defined according to some internal paperclip model). Humans using paperclips is negative by that standard (they occasionally break them, and by not being paperclips they represent untapped resources for more paperclips). <br><br>It is this kind of slip of meaning that makes a lot of verbal AI arguments mistaken. Think of this as software code maximizing something, not as a human-like mind that cares and reveres. Yes, the *right* utility function might indeed producing caring behavior and maybe also revering mental states. But *most* utility functions that maximize the number of paperclips do not. Indeed, trying to write a caring utility function is way harder than a mere maximizer, so unless the programmer was strongly motivated to try from the start the most likely outcome is one of the other utility functions. <br><br>Writing your ethical system as computer executable code (or code that reliably learns it) is a nontrivial challenge. Writing an explicit ethical system that doesn't have crazy side effects that render it contradictory to your intentions is even harder. <br><br>Saying it is OK that we don't know what will happen is a bit like pouring out chemicals at random from your lab into the water supply: maybe they will make a cure for cancer. But very probably it is not a wise move.<br><br><br>Anders Sandberg,
Future of Humanity Institute
Philosophy Faculty of Oxford University</body></html>