<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 2016-02-26 03:15, John Clark wrote:<br>
</div>
<blockquote
cite="mid:CAJPayv3Ec_5dQ==rUCZLXS8JEmDK+eoq_=29KEVr7gxvrr+4Yg@mail.gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif"><span
style="font-family:arial,sans-serif">On Thu, Feb 25, 2016 at
4:47 PM, Anders Sandberg </span><span dir="ltr"
style="font-family:arial,sans-serif"><<a
moz-do-not-send="true" href="mailto:anders@aleph.se"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:anders@aleph.se">anders@aleph.se</a></a>></span><span
style="font-family:arial,sans-serif"> wrote:</span><br>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif;display:inline">
> </div>
A utility-maximizer in a complex environment will not
necessarily loop</div>
</blockquote>
<div><br>
</div>
<div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif;display:inline"><font
size="4">If there is a fixed goal in there that can
never be changed then a infinite loop is just a matter
of time, and probably not much time.</font></div>
<font size="4"> </font></div>
</div>
</div>
</div>
</blockquote>
<br>
I wonder what you mean by an "infinite loop". Because clearly you
mean something different from "will repeat the same actions again
and again" given my example. <br>
<br>
My *suspicion* is that you mean "will never do anything creative",
which is a very different thing. But even there we have
counterexamples: evolution is a fitness-maximizer, and genetic
programming allows you to evolve highly creative solutions to
problems (consider the creatures of Karl Sims, for example). It is
trivial in principle (another thing in practice) to embed such
algorithms in a utility maximizer agent. <br>
<br>
<blockquote
cite="mid:CAJPayv3Ec_5dQ==rUCZLXS8JEmDK+eoq_=29KEVr7gxvrr+4Yg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote"><br>
</div>
</div>
<div class="gmail_quote">
<div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif;display:inline"><font
size="4">Real minds don't get into infinite loops thanks
to one of Evolutions greatest inventions, boredom.
Without a escape hatch a innocent sounding request could
easily turn the mighty multi billion dollar AI into
nothing but a space heater. <br>
</font></div>
</div>
</div>
</div>
</blockquote>
<br>
You know boredom is trivially easy to implement in your AI? I did it
as an undergraduate. <br>
<br>
(Essentially it is a moderately fast updating value function
separate from the real value function, which you subtract from it
when doing action selection. That makes the system try new actions
if it repeats the same actions too often. )<br>
<br>
<blockquote
cite="mid:CAJPayv3Ec_5dQ==rUCZLXS8JEmDK+eoq_=29KEVr7gxvrr+4Yg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote"><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif;display:inline">
> </div>
Try to crash Siri with a question. <br>
</div>
</blockquote>
<div><br>
</div>
<div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif"><font
size="4">You can't crash Siri because Siri doesn't
have a fixed goal, certainly not the fixed goal of
"always do what a human tells you to do no matter
what". So if you say "Siri, find the eleventh prime
number larger than 10^100^100" she will simply say
"no, I don't want to". </font></div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
Hmm. You seem to assume the fixed goal is something simple,
expressible as a nice human sentence. Not utility maximization over
an updateable utility function, not as trying to act optimally over
a space of procedures (which is what I guess Siri is like).<br>
<br>
One of the biggest problems in AI safety is that people tend to make
general statements for or against in human language that actually do
not show anything rigorously. This is why MIRI and some of the
FHIers have gone all technical: normal handwaving does not cut it,
neither for showing risk nor for showing safety. <br>
<br>
<br>
<pre class="moz-signature" cols="72">--
Dr Anders Sandberg
Future of Humanity Institute
Oxford Martin School
Oxford University
</pre>
</body>
</html>