[ExI] 'Friendly' AI won't make any difference

[Anders Sandberg]

On 2016-02-28 04:40, TimTyler wrote:
>
> You can loop forever without ever returning to a previous state, though:
>
> 10 LET UNBOUNDED INTEGER i = 0;
> 20 i = i + 1;
> 30 GOTO 20;

Yeah, that was stupid of me.

Still, the general point still stands: despite the halting theorem it is 
quite possible to detect large categories of infinite loops 
automatically (this is something real code validation systems actually 
manage - they have no problem proving the above will not halt).

Now, in AI we do not usually want the agent to halt (for a 
question-answer system this is the goal, but not for a robot). Instead 
we want it to behave indefinitely, yet not get stuck in a maladaptive 
behavioral loop. The interesting thing there is that one can prove stuff 
about the ergodicity of behavior in some cases like reinforcement 
learners, showing they will explore their space and if there is a better 
action pattern they will eventually find it. The real problem is usually 
that that convergence is impractically slow.

-- 
Dr Anders Sandberg
Future of Humanity Institute
Oxford Martin School
Oxford University