<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
On 2016-02-25 18:39, John Clark wrote:<br>
<blockquote
cite="mid:CAJPayv3Cu1sVNWxHzUXh6Kqe5dBkNOA0a_6djM219+w_7Zwaqg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote"><font size="4">There are indeed
vested interests</font>
<div><font size="4">
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif;display:inline">
but it wouldn't matter even if there weren't, </div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif;display:inline">there
is no way the friendly AI (aka slave AI) idea could
work under any circumstances. You just can't keep
outsmarting something far smarter than you are
indefinitely</div>
</font></div>
</div>
</div>
</div>
</blockquote>
<br>
Actually, yes, you can. But you need to construct utility functions
with invariant subspaces - that is, there are mathematically
provable solutions (see the work of Stuart Armstrong:
<a class="moz-txt-link-freetext" href="http://www.fhi.ox.ac.uk/utility-indifference.pdf">http://www.fhi.ox.ac.uk/utility-indifference.pdf</a> and sequels). Their
descriptions do not simplify to everyday descriptions people like to
use when making claims like the above. <br>
<br>
Are they practical? Current methods are not workable. But that does
not tell us much about the space of other possibilities; it would be
nice if the sceptics were to try to turn their claims into crisp
theorems - that would advance the field too. <br>
<br>
<pre class="moz-signature" cols="72">--
Anders Sandberg
Future of Humanity Institute
Oxford Martin School
Oxford University</pre>
</body>
</html>