[ExI] Unilateralist

[Anders Sandberg]

On 2012-12-27 08:10, Daniel Shown wrote:
> "The best solution would be to have all people involved get together and
> pool their knowledge, making a joint decision"
>
> How certain of this are you?

Aumann agreement theorem and Bayesian rationality. A perfect Bayesian 
agent who receives new information will act better or equal to its prior 
state, since either the information is useful or it is not, in which 
case it is ignored[*]. Rational agents with the same priors who share 
even a small amount of information will also come to agree completely 
with each other: http://wiki.lesswrong.com/wiki/Aumann's_agreement_theorem

In *practice* this is less effective. Perfect Bayesian agents are 
computationally too expensive for human minds to emulate, and we know 
from practice that even well-meaning fairly rational people do disagree 
with each other. But if the agents are not willfully incompetent they 
can still do significantly better than naive agents (or even fairly 
clever rational but isolated agents).


For example, suppose the true value of the action is a Gaussian 
distributed random number, and each agent gets a noisy signal (the value 
plus some Gaussian noise with the same variance). If one agent decides 
to act they all get rewarded the true value, otherwise zero. In the 
omniscient case where they somehow magically see through the noise they 
will act just when the value is positive, but in reality they will slip 
occasionally and get less. We want to reduce this performance loss.

Roughly speaking, the expected performance loss of a single agent 
compared to a group of 5 naive agents (there is almost always one agent 
that thinks a negative value is positive) is about half: they miss out 
half as much as the group (and larger groups do even worse, of course). 
Using a Bayesian threshold setting calculation (no communication 
involved) halves the losses again if all agents in the group use it. If 
they do a majority vote (just signalling yeas and nays) the losses are 
halved again. If they share the noisy estimates of the true value they 
have and do a maximum likelihood estimation (which in this case is a 
simple mean) they get a slight improvement over majority voting, but it 
is not huge. It seems that they cannot improve their performance beyond 
this, since there simply is no more data to process. But by now they are 
surprisingly close to the omniscient case performance.


So our advice for unilateralist curse situations is: 1) if possible, 
talk and set up a joint decision. 2) if talking is not possible, 
calculate how a rational agent should have solved the situation 
(including uncertainty about the other agents and their abilities) and 
act like that, 3) if that is too complex and you can just randomly 
select a single agent to act, do that (yes, sometimes the rational 
choice is to flip a coin to decide whether to act even when you think 
the action is good). 4) if that cannot be done either, try to defer to a 
group consensus (real or imaginary) about this type of action rather 
than striking out unilaterally.

I find it intellectually enjoyable to see that our paper leads to a 
conclusion I intuitively do not like: deferring to consensus rather than 
striking out gloriously individually.


[*] Yes, there are things like anti-predictable sequences and negative 
learning (see
http://www.princeton.edu/step/people/faculty/michael-oppenheimer/recent-publications/climatechange2008.pdf 
), but a full Bayesian agent is immune to them. Which is why they don't 
exist in practical reality.

-- 
Anders Sandberg
Future of Humanity Institute
Oxford University