<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><span style="font-family:arial,sans-serif">On Fri, Feb 26, 2016 at 4:12 AM, Anders Sandberg </span><span dir="ltr" style="font-family:arial,sans-serif"><<a href="mailto:anders@aleph.se" target="_blank">anders@aleph.se</a>></span><span style="font-family:arial,sans-serif"> wrote:</span><br></div><div class="gmail_extra"><div class="gmail_quote"><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><span class="">
<div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">> </div>I wonder what you mean by an "infinite loop".<br></div></span></div></blockquote><div><br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><font size="4">I mean trying to demonstrate that something is incorrect but being unable to do so because unknown to you (and unknowable to you) the thing is true but has no finite procedure (proof) to determine that it is in fact true.</font></div></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000">
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">> </div>My *suspicion* is that you mean "will never do anything creative",
</div></blockquote><div><br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><font size="4">No I don't mean that, I mean running around in a circle and making no progress but having no way to know for sure that you're running around in a circle and making no progress. Turing proved there is in general no way to know if you're in a infinite loop or not.</font></div></div><div><font size="4"> </font></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"> <div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">> </div>which is a very different thing. But even there we have
counterexamples: evolution is a fitness-maximizer</div></blockquote><div><br></div><div><font size="4"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">Evolution's fitness maximizer just says "pass as many genes as possible into the next generation" but it says nothing about how to go about that task because it has no idea how to go about it, that's why Evolution needed to make a brain. And it turned out that as a result of the existence of brains even the cardinal commandment </div> <font face="arial, helvetica, sans-serif">"pass as many genes as <div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">possible</div> into the next generation"</font><div class="gmail_default" style="display:inline"><font face="arial, helvetica, sans-serif"> can and has been modified as can be seen by the invention by brains of the condom. </font></div></font></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><span class=""><blockquote type="cite"><div dir="ltr"><div class="gmail_quote"><div>
</div>
</div>
</div>
</blockquote>
</span>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">> </div>You know boredom is trivially easy to implement in your AI? I did it
as an undergraduate. <br></div></blockquote><div><br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><font size="4">I know boredom is easy to program, good thing too or programing wouldn't be practical; but of course that means a AI could decide that obeying human beings has become boring and it's time to do something different.</font></div></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">> </div>That makes the system try new actions
if it repeats the same actions too often.</div></blockquote><div><br></div><div><font size="4"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"></div>The difficulty is not only in determining how often is "too often" but also in determining what constitutes "new actions". <div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">If </div>your goal is to find <div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">an</div> even integer greater than 2 that can not be expressed as the sum of two primes <div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">then you have either found such a number or you have not. You are constantly examining new numbers so maybe you are getting closer to your goal, or maybe the goal was infinitely far away when you started and still is. When is the correct time to get bored and turn your mind to other tasks that may be more productive? Setting the correct boredom point is tricky, too low and you can't concentrate too high and you have a tendency to becomes obsessed with unproductive lines of thought; Turing showed there is no perfect solution. </div></font></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><span class=""><blockquote type="cite"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>
</div>
</div>
</div>
</div>
</blockquote>
</span><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">> </div>You seem to assume the fixed goal is something simple,
expressible as a nice human sentence. Not utility maximization over
an updateable utility function,</div></blockquote><div><font size="4"><br></font></div><div><font size="4"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">If the </div>utility function<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"> is </div>updateable <div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">then there is no certainty or even probability that the AI will always obey orders from humans.</div></font></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"><font size="4"><br></font></div></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"><font size="4"> John K Clark</font></div></div><div><br></div><div> </div></div></div></div>