[ExI] Machines of Loving Grace

[Ben Zaiboc]

On 12/10/2024 19:41, BillK wrote:
> Dario Amodei, the CEO of Anthropic (the company that has developed
> Claude AI) has written a paper that discusses the benefits that AI
> could bring to the world over the next 5 to 10 years. I thought it was
> rather impressive - assuming that the AI effects are beneficial.
> Worth a read!
>
> BillK
>
> <https://darioamodei.com/machines-of-loving-grace>

Well worth reading, I agree.

On the whole I enjoyed it, but thought there was one glaring omission: 
Self-awareness (aka 'consciousness').

I get the impression that he prefers the term "powerful AI" over "AGI" 
because it gives him a chance to dodge this issue. But I don't think you 
can dodge it when talking about entities with the level of intelligence 
that he supposes. Especially as he says he's concerned about the dangers 
of AI.

And no, the issue of Self-awareness in AI doesn't fall under the heading 
of 'science-fiction baggage', for the following reason:

When an AI is capable of the kind of problem-solving he mentions, 
there's no avoiding the fact that it must develop and use some kind of 
Theory Of Mind ('TOM'). To interact sensibly with someone, you need to 
create a mental model that represents them, so you can remember who they 
are, distinguish them from other agents you interact with, assign 
preferences and personality characteristics to them, make predictions 
about them, and their interactions with other agents that you have 
models of (She thinks that he thinks, etc.), and so-on. Do that, and 
it's not long before your TOM system has to include another model, of 
yourself. Et voila: Self-awareness, by definition.

(If you disagree with my equating Self-awareness and Consciousness, Ok, 
just forget about the C-word. We can manage fine without it).

This intelligent biological machine remembers it happening to me when I 
was young. It was a real "AHA!!" moment. For me, at least, it was like 
waking up from a dream. I imagine it would have a similar impact on 
intelligent non-biological machines.

Current AI systems saying things like "I think that...", really don't. 
It's fake, probably pre-programmed as a way to present information to 
people, or maybe just picked up from training data. Without TOM, you 
have no idea who 'you' are. The concept doesn't exist. But when you 
start using TOM, I don't see how it's not inevitable that you develop 
some kind of self-awareness, probably quite soon.

Given all this, I find it bizarre that he can talk about such advanced, 
powerful AIs without addressing the idea that they will inevitably 
require TOM to function well, and so will inevitably 'wake up' and 
become self-aware.

I know that that opens a totally separate, huge can of worms that he 
probably can't even begin to address in that article, but it does need 
to be addressed, and prepared for.

Is there a way to avoid this? I can understand, if not agree with, 
people who want to.

I can't think of any way to take advantage of TOM without it leading to 
self-awareness*. People cleverer than me (I know there are many!) might 
be able to.

In any case, I think any avoidance would be only temporary, and we 
really do need to anticipate this happening at some point, in the 
probably-nearer-than-we-think future.


Ben

* One thing I just thought of was being able to control how strong, or 
detailed, the mental models of the various agents are. The problem there 
is that if you make the system incapable of making detailed models (so 
making sure its self-model is fuzzy), then you're deliberately crippling 
its effectiveness. Also, it might not make any real difference. It's 
still going to have a 'Me', just a vaguer one.

-- 
I'm a big fan of intelligent design.  I think it's a great idea, and we should proceed with it as soon as possible.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20241017/9db8a4b0/attachment.htm>