[ExI] the Bluebird scenario

[Anders Sandberg]

On 2015-09-24 12:24, spike wrote:
>
> Of all the AI discussion here, I don’t recall any theoretical AI which 
> emerged and subsequently reached a long-term equilibrium after having 
> taken over some modest segment or task.  We need a map of all AI 
> possibilities to do valid Baysian statistics, and the Bluebird 
> scenario is one of them.
>
>

It depends on the goal structure. Maximizers try to maximize some 
utility function, and generically that tends to end badly. It is not 
just paperclip maximizers that steamroll the universe: the agent ordered 
to make one (1) hamburger will also try to ensure that the probability 
of success is as high as possible. Which might include adding 
surveillance and armor to ensure that the hamburger really *is* there 
and will not be stolen... in fact, let's get rid of all other agents 
that could interfere to pre-empt any theft...

Satisficers seem much better: they will be happy with reaching a certain 
goal well enough. However, as some of my colleaues pointed out, they can 
start misbehaving too by becoming maximizers: 
http://lesswrong.com/lw/854/satisficers_want_to_become_maximisers/

Now, humans are not well modelled as utility maximizers or satisficers 
in a lot of situations. We have a mess of goals, and some actions are 
not even goal-oriented. It is not too hard to make AIs just as hopeless: 
just connect a random neural network to a robot and set it off - but 
obviously humans are a bit better at doing something sensible. But in 
that space of networks there are doubtless some that are 
human/Bluebird-like. It just seems to me that they have a very small 
measure compared to the "wobble around in pointless circles" and 
"maximize X" agents.

There is a tricky interaction between having goals and being 
intelligent: intelligence can be defined as the ability to achieve goals 
in general circumstances, so to be intelligent you need to have some 
kind of goals. But clearly some goals are special: even I can write an 
AI whose goal is to not do anything, but I think we can agree it's 
intelligence is not interesting. So interesting intelligence requires 
nontrivial goals - so in that space of minds there will be some 
correlation between being goal-oriented and intelligent. But some agents 
may not have an entirely goal-oriented architecture, yet be pretty good 
at achieving what we call goals.

Mapping mindspace is fun! There are *weird* corners out there...

-- 
Anders Sandberg
Future of Humanity Institute
Oxford Martin School
Oxford University

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20150924/93350291/attachment.html>