[ExI] Fermi Paradox and Transcension

[Anders Sandberg]

On 05/09/2012 15:59, Tim Halterman wrote:
>> Two great intellects meet.  Advancement (perhaps omnipotence) is
>> within their grasp, that means however one must have power (to gain
>> their perspective or place in the universe if nothing else) over the
>> other.  Both could stay stagnant but that would not be advancement.
>> Sharing sounds nice but even handing another all of your knowledge
>> will not give the unique perspective an individual will have on the
>> universe (I believe if you were to give your perspective of the
>> universe to another being you would cease to be separate and would
>> become just one being).  What are rational conclusions to this
>> scenario?

Treat it as a game theory problem. Doing nothing has value zero. Both 
cooperating has mutual reward R, cooperating with the other who cheats 
has a bad outcome F (F<R), cheating on the other has value C (>0). You 
decide to cooperate with probability p, and can assume the other (being 
entirely symmetric) will arrive at the same p. You agree to flip a coin 
to determine who will be in whose power.

The value of cooperating will be p*R + (1-p)(F) if you are the one 
giving up power and R if you are the one in power. So the expected value 
of being nice is (1/2)((1+p)*R + (1-p)F). If this is bigger than C, the 
value of defecting while in power is C, then there exists a cooperative  
strategy. That is, if R+F + p*(R-F) > 2C the beings will be nice. The 
biggest value it can have is for p=1, 2R. So the condition is: 
cooperation if R>C.


-- 
Anders Sandberg,
Future of Humanity Institute
Philosophy Faculty of Oxford University