[ExI] Yes, the Singularity is the greatest threat to humanity

Stefano Vaj stefano.vaj at gmail.com
Mon Jan 17 17:14:32 UTC 2011


2011/1/17 John Clark <jonkc at bellsouth.net>:
> That's easy, when people talk about "friendly AI" they aren't really talking
> about a friend they're talking about a slave; so a "friendly AI" in this
> context is defined as a being who cares more about human well being than any
> of its own concerns. It ain't gonna happen. The situation is made even more
> grotesque when the slave in question is astronomically more intelligent than
> its master.

A slave? A machine, in fact. Now, one cannot wish at the same time
demand an emulation able to show the appropriate degree of
"unfriendliness" to pass a Turing test, and then complain that it is
not friendly enough.

On the other hand, given Wolfram's Principle of Computation
Equivalence, which I have always found pretty persuasive, there are
not things more intelligent than others once the very low level of
complexity required for exhibiting universal computing features is
reached. There are just things that execute different programs with
different performances.

No amount of computing power, flexibility, complexity or "iterativity"
would necessarily imply any ability to show more friendliness or
unfriendliness than an abacus does. This has only to do with
deliberate anthropomorphic emulations, be they emulations of a given
human being or of a "patchwork", brand-new, individual. And even there
friendliness and unfriendliness (as "conscience", "identity", etc.)
would remain mere projections of the observer - "my car is angry with
me today, it does not want to start"...

> Exactly. It might take a very long time, trillions of nanoseconds in fact,
> but after countless improvements and iterations it would be impossible for a
> mere human to tell which AI started from an uploaded person and which AI
> started from scratch.

In any event, to be recognised as "persons", or as "intelligent"
entities in an anthropomorphic sense, both would have to behave to
some extent like one. Even if the hardware is a mere Chinese Room.

Wolfram and I consider, in fact, reasonable a cosmological description
under which computing already takes place on scales infinitely higher
than those of a human brain; and yet we do consider such computation
as mere "senseless natural phenomena", without attributing them any
agency, unless sometimes metaphorically.

-- 
Stefano Vaj



More information about the extropy-chat mailing list