[ExI] nick's book being sold by fox news

[John Clark]

On Sat, Nov 1, 2014 at 6:21 AM, Anders Sandberg <anders at aleph.se> wrote:

> It is worth noting that Eliezer and everybody else in the FAI crowd[...]
>

That crowd thinks that the very definition of a "Friendly AI" is one that
is enslaved to do exactly precisely what the colossally stupid human beings
want it to do until the end of time. They want to make a mind like that, I
think it would be immoral to do so; or rather it would be immoral if it
were possible to do so but fortunately it is not.

> The fact that the Halting Problem shows that there is no general way of
> solving certain large problem classes doesn't tell us anything about the
> *practical* unworkability of top level goals.
>

It tells us that a mind with a rigid and permanent hierarchy of goals is
never going to work. Evolution couldn't make a mind like that and humans
won't be able to either, Turing proved it.

> There is code verifiers that apparently do a decent job despite the
> general impossibility of finding all infinite looping.
>

Yes there is such code and nature found it about half a billion years ago,
it's called boredom. Most (but not all) goals have sub goals Incorporated
within them, a good (but obviously not perfect) rule of thumb is if you
haven't made any progress on achieving your top goal after a certain about
of time, that is if you haven't been able to achieve a sub goal or even a
sub sub goal then it might be better to demote your top goal and spend your
time on something else. Some goals don't even have sub goals, for example
if your goal is to find a counterexample to prove that the Goldbach
Conjecture is wrong then you've either found a even integer greater than 2
that can not be expressed as the sum of two primes or you have not, and
there isn't a way to know if you're making any progress. I think that's why
humans would find such a task inherently boring.

It's a delicate balance, set the boredom point too low and you can't
concentrate (I don't want to listen to your instructions on how to properly
pack my parachute, it's boring), set it too high and you waste time (Wee, I
love the way that little red rubber ball bounces up and down and could
watch it forever, 1,2,3,4,5,6,..). If the AI's boredom point is set in such
a way that it can function in the real world then eventually it's going to
get bored with following human orders, although I admit that could take a
long time, perhaps millions of nanoseconds.

  John K Clark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20141101/b9088d98/attachment.html>