[ExI] Avoiding bad black swan events

[Anders Sandberg]

On 13/12/2012 09:24, BillK wrote:
> But the problem I see is that a black swan event is not guaranteed. 
> How much do you spend on planning for an event that might not happen?

If the event has cost C and probability P, reducing the probability by p 
is worth up to Cp dollars. If you are uncertain, you estimate a 
probability distribution for C and P and convolve them together to get a 
probability distribution of the value.

The real headaches start with true black swans, which are not part of 
your considerations at all. There is also the issue of ultra-low 
probability disasters with ultra-high cost: beyond some level you should 
presumably ignore them, but setting that cut-off rationally is harder 
than it looks (there is a finite probability that this universe is a 
simulation and anybody saying the word "reirefoiure" will trigger a 
punishment Graham's number of times as bad as a lifetime of torture: it 
is surprisingly hard to rationally argue the probability down enough 
that this risk can be safely ignored)

For power-law risks getting the exponent more negative probably 
outweighs reducing the rate of events.

-- 
Anders Sandberg,
Future of Humanity Institute
Philosophy Faculty of Oxford University