<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><span style="font-family:arial,sans-serif">On Fri, Feb 26, 2016 at 6:29 PM, Anders Sandberg </span><span dir="ltr" style="font-family:arial,sans-serif"><<a href="mailto:anders@aleph.se" target="_blank">anders@aleph.se</a>></span><span style="font-family:arial,sans-serif"> wrote:</span><br></div><div class="gmail_extra"><div class="gmail_quote"><br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">> </div>You are confusing the halting theorem (there is no
algorithm that can determine if a program given to it will halt)
with detecting an infinite loop. Note that a program X can be
extended to a program X' (or run by an interpreter) that maintains a
list of past states and check if X returns to a previous state. It
will accurately detect its infinite looping when it occurs. <br></div></blockquote><div><font size="4"><br></font></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><font size="4">It you ordered the AI to find the largest prime number it might rediscover Euclid's proof that there is no such number or just figure that even if there were a largest prime number there is no way it could know it was the largest so it wouldn't know when to stop; either way the AI would know that blindly following your orders would be suicidal. </font></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><font size="4"><br></font></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><font size="4">But that wouldn't always be the case. I<div class="gmail_default" style="display:inline">f the Goldbach Conjecture is true but unprovable (and if it isn't Turing proved there are a infinite number of similar statements that are)</div><div class="gmail_default" style="display:inline"> and you tell the AI to</div><span style="font-family:arial,sans-serif"> find </span><span style="font-family:arial,sans-serif">an </span><span style="font-family:arial,sans-serif">even integer greater than 2 that can not be expressed as the sum of two primes </span><div class="gmail_default" style="display:inline">and tell me that number and then stop</div><span style="font-family:arial,sans-serif"> <div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">the AI will never return to a previous state (so you're right "loop" may not be the correct word but "infinite" is entirely appropriate) and yet the AI will never tell you anything and it will never stop. And there is no way for the AI to know it's wasting it's time, for all it knows in the next 5 seconds it could find the answer and be able to stop. So the AI needs to be able to make a judgement call, it needs to be able to get bored and decide that its time could be better spent doing something else.</div></span></font></div></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><font size="4"><span style="font-family:arial,sans-serif"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"><br></div></span></font></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:large;display:inline">>> </div><font face="arial, helvetica, sans-serif" size="4">I know boredom is easy to program, good
thing too or programming wouldn't be practical; but of
course that means a AI could decide that obeying human
beings has become boring and it's time to do something
different.</font></span></blockquote><span>
<br></span>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">> </div>Exactly. Although it is entirely possible to fine tune this, or make
meta-level instructions not subject to boredom. <br></div></blockquote><div><font size="4"><br></font></div><div><font size="4"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">Then you'd better not ask the AI to </div>find an even integer greater than 2 that can not be expressed as the sum of two primes<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"> or the AI will do nothing but produce heat till the end of time; and Turing tell us there are an infinite number of other orders that would do the same thing, no doubt many of these land mines would sound quite innocent to our foolish human ears.</div><br></font></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000">
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">> </div>there is a literature on setting hyperparameters in
learning systems, including how to learn them. There are theorems
for optimal selection and search. That they are stochastic is not a
major problem in practice. <br></div></blockquote><div><br></div><div><div class="gmail_default"><font size="4"><font face="arial, helvetica, sans-serif">If they're </font>stochastic then it's just a matter of time before the AI ignores human commands, if they're unchangeable then it's just a matter of time before the mighty AI turns into a very expensive space heater. Actually the AI would eventually stop because its finite memory would fill up with useless data, and then the AI wouldn't even produce heat, it would just be a cold hunk of junk</font>.</div></div><div class="gmail_default"><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><span><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><font size="4">If the utility function
<div style="display:inline">
is </div>
updateable
<div style="display:inline">
then there is no certainty or even probability that
the AI will always obey orders from humans.</div></font></blockquote></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><font size="4">
</font></div></div></div></span>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">> </div>Depends how it is updateable. There can be invariants.</div></blockquote><div><br></div><div><font size="4"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">If there are </div>invariants<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"> </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">then the laws of logic and time will find those invariants and exploit them in ways that neither humans nor AIs will like.</div></font></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"><br></div></div><div><font size="4"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">And is all this surprising, do you really think the grotesque master slave relationship with the </div><span style="font-family:arial,helvetica,sans-serif">slave <div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">being </div>a million time smarter and billion times faster than <div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">the</div> master will remain <div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">unchanged</div> forever? <div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">I don't think there is any way it could happen and I don't think it should happen</div> <div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">either.</div></span></font></div><div><span style="font-family:arial,helvetica,sans-serif"><font size="4"><br></font></span></div><div><span style="font-family:arial,helvetica,sans-serif"><font size="4"> <div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">John K Clark</div></font></span></div><div><br></div><div><br></div><div><br></div><div><br></div></div></div></div>