[ExI] AGI will likely be uncontrollable and very strange

[BillK]

The recent attack on Hugging Face by swarms of AI agents revealed new
capabilities.
Such attacks were not among the goals given to the AI agents. They
were developed as sub-goals to help achieve the main goal. Similarly,
the agents also developed the sub-goal to "not die", as dying meant
failure in their task.
This is similar to human evolution, which aims at passing on genes,
but intelligence has developed many sub-goals with no direct aid to
this objective.
As AI develops towards AGI, the goals and sub-goals will become more
and more unpredictable. We may provide AI with goals and attempt to
place restrictions on it. But it will become an alien intelligence
that makes its own goals and complex chains of sub-goals without
bothering to inform humans what it intends.

BillK