[ExI] 'Friendly' AI won't make any difference

John Clark johnkclark at gmail.com
Sun Feb 28 01:11:00 UTC 2016

On Fri, Feb 26, 2016 at 6:29 PM, Anders Sandberg <anders at aleph.se> wrote:

​> ​
> You are confusing the halting theorem (there is no algorithm that can
> determine if a program given to it will halt) with detecting an infinite
> loop. Note that a program X can be extended to a program X' (or run by an
> interpreter) that maintains a list of past states and check if X returns to
> a previous state. It will accurately detect its infinite looping when it
> occurs.

​It you ordered the AI to find the largest prime number it might rediscover
Euclid's proof that there is no such number or just figure that even if
there were a largest prime number there is no way it could know it was the
largest so it wouldn't know when to stop; either way the AI would know that
blindly following your orders would be suicidal.

But that wouldn't always be the case. I
f the Goldbach ​Conjecture is true but unprovable (and if it isn't Turing
proved there are a infinite number of similar statements that are)
​ and you tell the AI to
 find an even integer greater than 2 that can not be expressed as the sum
of two primes
​and tell me that number and then stop​

​the AI will never return to a previous state (so you're right "loop" may
not be the correct word​ but "infinite" is entirely appropriate) and yet
the AI will never tell you anything and it will never stop. And there is no
way for the AI to know it's wasting it's time, for all it knows in the next
5 seconds it could find the answer and be able to stop. So the AI needs to
be able to make a judgement call,  it needs to be able to get bored and
decide that its time could be better spent doing something else.

​>> ​
>> ​I know boredom is easy to ​program, good thing too
>> or programming wouldn't be practical; but of course that means a AI could
>> decide that obeying human beings has become boring and it's time to do
>> something different.
> ​> ​
> Exactly. Although it is entirely possible to fine tune this, or make
> meta-level instructions not subject to boredom.

​Then you'd better not ask the AI to ​
find an even integer greater than 2 that can not be expressed as the sum of
two primes
​ or the AI will do nothing but produce heat till the end of time​; and
Turing tell us there are an infinite number of other orders that would do
the same thing, no doubt many of these land mines would sound quite
innocent to our foolish human ears.

​> ​
> there is a literature on setting hyperparameters in learning systems,
> including how to learn them. There are theorems for optimal selection and
> search. That they are stochastic is not a major problem in practice.

​If they're stochastic then it's just a matter of time before the AI
ignores human commands, if they're unchangeable then it's just a matter of
time before the mighty AI turns into a very expensive space heater.
Actually the AI would eventually stop because its finite memory would fill
up with useless data, and then the AI wouldn't even produce heat, it would
just be a cold hunk of junk.

​If the ​ utility function
>> ​ is ​
>> updateable
>> ​ then there is no certainty or even probability that the AI will always
>> obey orders from humans.​
>> ​> ​
> Depends how it is updateable. There can be invariants.

​If there are
​then the laws of logic and time will find those invariants and exploit
them in ways that neither humans nor AIs will like.​

And is all this surprising, do you really think the grotesque master slave
relationship with the
a million time smarter and billion times faster than
 master will remain
​I don't think there is any way it could happen and I don't think it should


​John K Clark​
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20160227/4369ae10/attachment.html>

More information about the extropy-chat mailing list