[ExI] Seemingly Conscious AI Is Coming

[Ben Zaiboc]

On 17/09/2025 14:40, BillK reported:
> Seemingly Conscious AI Is Coming
> Sep 15, 2025 Mustafa Suleyman
>
> Debates about whether AI truly can be conscious are a distraction.
> What matters in the near term is the perception that they are – and
> why the temptation to design AI systems that foster this perception
> must be resisted.
>
> <https://www.project-syndicate.org/commentary/seemingly-conscious-ai-urgent-threat-tech-industry-must-address-by-mustafa-suleyman-2025-09>
> Quote:
> An SCAI would be capable of fluently using natural language,
> displaying a persuasive and emotionally resonant personality. It would
> have a long, accurate memory that fosters a coherent sense of itself,
> and it would use this capacity to claim subjective experience (by
> referencing past interactions and memories). Complex reward functions
> within these models would simulate intrinsic motivation, and advanced
> goal setting and planning would reinforce our sense that the AI is
> exercising true agency.
>
> All these capabilities are already here or around the corner. We must
> recognize that such systems will soon be possible, begin thinking
> through the implications, and set a norm against the pursuit of
> illusory consciousness.
> ------------------------
>
> This is the "consciousness" problem becoming real.
> If an AI successfully appears to be conscious, how can we test whether it is
> a really conscious creation?
> BillK

You decide which features a conscious system has, then see if the AI 
system has those same features. Here's a tentative list:

capable of fluently using natural language,
displaying a persuasive and emotionally resonant personality
a long, accurate memory that fosters a coherent sense of self
ability to reference past interactions and memories
Complex reward functions within these models that implement intrinsic motivation
advanced goal setting and planning

There will probably be more (my list in a recent post may be relevant as 
well, I can see some overlap)

I actually know some people who display these characteristics. Normally, 
though, I just use 'Duck theory", and don't worry if  they are really 
conscious or not. It doesn't make any practical difference, after all.

We could, though, apply this test to Mustafa Suleyman, and use it to 
decide if he is truly conscious or 'just simulating it'. Presumably, if 
we decided the latter, he would agree that he no longer has any human 
rights, and we could just turn him off without any qualms. One factor 
that weighs against him is his apparent inability to see the silliness, 
or at least the irrelevance, of the statement "a simulation of a storm 
doesn’t mean it rains in your computer". I'm inclined to think he has no 
consciousness of what he's saying.

-- 
Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20250918/ba23dba3/attachment.htm>