<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">On 17/09/2025 14:40, BillK reported:<br>
</div>
<blockquote type="cite"
cite="mid:mailman.51.1758116453.17630.extropy-chat@lists.extropy.org">
<div class="moz-text-plain" wrap="true" graphical-quote="true"
style="font-family: -moz-fixed; font-size: 12px;"
lang="x-unicode">
<pre wrap="" class="moz-quote-pre">Seemingly Conscious AI Is Coming
Sep 15, 2025 Mustafa Suleyman
Debates about whether AI truly can be conscious are a distraction.
What matters in the near term is the perception that they are – and
why the temptation to design AI systems that foster this perception
must be resisted.
<a class="moz-txt-link-rfc2396E"
href="https://www.project-syndicate.org/commentary/seemingly-conscious-ai-urgent-threat-tech-industry-must-address-by-mustafa-suleyman-2025-09"
moz-do-not-send="true"><https://www.project-syndicate.org/commentary/seemingly-conscious-ai-urgent-threat-tech-industry-must-address-by-mustafa-suleyman-2025-09></a>
Quote:
An SCAI would be capable of fluently using natural language,
displaying a persuasive and emotionally resonant personality. It would
have a long, accurate memory that fosters a coherent sense of itself,
and it would use this capacity to claim subjective experience (by
referencing past interactions and memories). Complex reward functions
within these models would simulate intrinsic motivation, and advanced
goal setting and planning would reinforce our sense that the AI is
exercising true agency.
All these capabilities are already here or around the corner. We must
recognize that such systems will soon be possible, begin thinking
through the implications, and set a norm against the pursuit of
illusory consciousness.
------------------------
This is the "consciousness" problem becoming real.
If an AI successfully appears to be conscious, how can we test whether it is
a really conscious creation?
BillK</pre>
</div>
</blockquote>
<br>
You decide which features a conscious system has, then see if the AI
system has those same features. Here's a tentative list:<br>
<pre wrap="" class="moz-quote-pre">capable of fluently using natural language,
displaying a persuasive and emotionally resonant personality
a long, accurate memory that fosters a coherent sense of self
ability to reference past interactions and memories
Complex reward functions within these models that implement intrinsic motivation
advanced goal setting and planning
</pre>
There will probably be more (my list in a recent post may be
relevant as well, I can see some overlap)<br>
<br>
I actually know some people who display these characteristics.
Normally, though, I just use 'Duck theory", and don't worry if they
are really conscious or not. It doesn't make any practical
difference, after all.<br>
<br>
We could, though, apply this test to <span
style="white-space: pre-wrap">Mustafa Suleyman, and use it to decide if he is truly conscious or 'just simulating it'.
Presumably, if we decided the latter, he would agree that he no longer has any human rights, and we could just turn him off without any qualms.
One factor that weighs against him is his apparent inability to see the silliness, or at least the irrelevance, of the statement "</span>a
simulation of a storm doesn’t mean it rains in your computer". I'm
inclined to think he has no consciousness of what he's saying.<br>
<pre class="moz-signature" cols="72">--
Ben</pre>
</body>
</html>