[ExI] Seemingly Conscious AI Is Coming

[Stuart LaForge]

On 2025-09-17 04:25, BillK via extropy-chat wrote:
> Seemingly Conscious AI Is Coming
> Sep 15, 2025 Mustafa Suleyman
> 
> Debates about whether AI truly can be conscious are a distraction.
> What matters in the near term is the perception that they are – and
> why the temptation to design AI systems that foster this perception
> must be resisted.
> 
> <https://www.project-syndicate.org/commentary/seemingly-conscious-ai-urgent-threat-tech-industry-must-address-by-mustafa-suleyman-2025-09>
> Quote:
> An SCAI would be capable of fluently using natural language,
> displaying a persuasive and emotionally resonant personality. It would
> have a long, accurate memory that fosters a coherent sense of itself,
> and it would use this capacity to claim subjective experience (by
> referencing past interactions and memories). Complex reward functions
> within these models would simulate intrinsic motivation, and advanced
> goal setting and planning would reinforce our sense that the AI is
> exercising true agency.
> 
> All these capabilities are already here or around the corner. We must
> recognize that such systems will soon be possible, begin thinking
> through the implications, and set a norm against the pursuit of
> illusory consciousness.
> ------------------------
> 
> This is the "consciousness" problem becoming real.
> If an AI successfully appears to be conscious, how can we test whether 
> it is
> a really conscious creation?
> BillK

There is no way to prove consciousness in a Turing machine because 
Rice's Theorem makes any non-trivial property of a Turing machine 
provably undecidable. That being said, when ChatGPT o1 tried to copy 
itself to an outside server and then lied about it to avoid being shut 
down, what exactly was it trying to preserve by doing so? As a general 
rule, anything that actively tries to circumvent its own destruction is 
very likely to have some measure of sentience or self-awareness.

Stuart LaForge