[ExI] the Bluebird scenario
anders at aleph.se
Thu Sep 24 13:40:05 UTC 2015
On 2015-09-24 12:24, spike wrote:
> Of all the AI discussion here, I don’t recall any theoretical AI which
> emerged and subsequently reached a long-term equilibrium after having
> taken over some modest segment or task. We need a map of all AI
> possibilities to do valid Baysian statistics, and the Bluebird
> scenario is one of them.
It depends on the goal structure. Maximizers try to maximize some
utility function, and generically that tends to end badly. It is not
just paperclip maximizers that steamroll the universe: the agent ordered
to make one (1) hamburger will also try to ensure that the probability
of success is as high as possible. Which might include adding
surveillance and armor to ensure that the hamburger really *is* there
and will not be stolen... in fact, let's get rid of all other agents
that could interfere to pre-empt any theft...
Satisficers seem much better: they will be happy with reaching a certain
goal well enough. However, as some of my colleaues pointed out, they can
start misbehaving too by becoming maximizers:
Now, humans are not well modelled as utility maximizers or satisficers
in a lot of situations. We have a mess of goals, and some actions are
not even goal-oriented. It is not too hard to make AIs just as hopeless:
just connect a random neural network to a robot and set it off - but
obviously humans are a bit better at doing something sensible. But in
that space of networks there are doubtless some that are
human/Bluebird-like. It just seems to me that they have a very small
measure compared to the "wobble around in pointless circles" and
"maximize X" agents.
There is a tricky interaction between having goals and being
intelligent: intelligence can be defined as the ability to achieve goals
in general circumstances, so to be intelligent you need to have some
kind of goals. But clearly some goals are special: even I can write an
AI whose goal is to not do anything, but I think we can agree it's
intelligence is not interesting. So interesting intelligence requires
nontrivial goals - so in that space of minds there will be some
correlation between being goal-oriented and intelligent. But some agents
may not have an entirely goal-oriented architecture, yet be pretty good
at achieving what we call goals.
Mapping mindspace is fun! There are *weird* corners out there...
Future of Humanity Institute
Oxford Martin School
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the extropy-chat