[ExI] AGI development and human extinction risk

[Ben Zaiboc]

On 21/03/2026 12:38, Keith Henson wrote:
> The driver for the mad rush for AI is that people think being involved
> early will make them wealthy.  Most of them are already wealthy,
> certainly to the point they will never miss a meal, so what drives
> Musk and company?
>
> I make the case that the evolved drive for wealth is open-ended.
> Those who had the trait of wealth accumulation in the past did well.
> As an example, consider farmers in the not-so-distant past who
> accumulated firewood for the winter.  It did not hurt their
> reproductive success to accumulate more, even a lot more, than was
> needed for winter.  However, when the exceptional winter came, they,
> and more importantly, their children survived while others froze.
> Their children (with this trait) occupied the farms of those who died
> in the cold.
>
> If you wonder where the open-ended drive for wealth comes from, this
> is it.  There has never been any reproductive disadvantage to having
> more wealth (or firewood).
>
> Not too many in the population have the combination of genes to
> accumulate great wealth, but there are certainly some.  Good or bad?
> I don't know.  For sure, Musk has caused some remarkable things to
> exist.


"There has never been any reproductive disadvantage to having more wealth (or firewood)."

I don't think that's exactly true. We (in common with many other mammals) have a tendency to punish those who take more than their fair share. This is probably older and stronger than the 'unlimited accumulation' trait, which might be unique to humans.

And there's probably a tendency to avoid this punishment by refraining from the unfair behaviour, even if you'd like to.

Maybe that's why only a small percentage of people do indulge in unlimited accumulation.

-- 
Ben