[ExI] Existential risk of AI

[BillK]

On Wed, 15 Mar 2023 at 02:02, Gadersd via extropy-chat
<extropy-chat at lists.extropy.org> wrote:
>
> I’m not sure where you got the idea that these models cannot reason.
> See [2212.10071] Large Language Models Are Reasoning Teachers (arxiv.org) for actual research into the matter.
> _______________________________________________


Fortunately, before release, these new LLM systems are being tested to
ensure they won't take over the world.
(Though, when the fatal version finally does arrive, the testing
itself may be enough to set the system loose).

<https://arstechnica.com/information-technology/2023/03/openai-checked-to-see-whether-gpt-4-could-take-over-the-world/>
Quotes:
With these fears present in the AI community, OpenAI granted the group
Alignment Research Center (ARC) early access to multiple versions of
the GPT-4 model to conduct some tests. Specifically, ARC evaluated
GPT-4's ability to make high-level plans, set up copies of itself,
acquire resources, hide itself on a server, and conduct phishing
attacks.

Christiano defended ARC's work with OpenAI, specifically mentioning
"gain-of-function" (AI gaining unexpected new abilities) and "AI
takeover":
I think it's important for ARC to handle the risk from
gain-of-function-like research carefully and I expect us to talk more
publicly (and get more input) about how we approach the tradeoffs.
This gets more important as we handle more intelligent models, and if
we pursue riskier approaches like fine-tuning.
------------------

Why do I have a funny feeling when he mentions 'gain-of-function'?


BillK