[extropy-chat] "3 Laws Unsafe" by the Singularity Institute

[Damien Broderick]

I regarded the robots speculatively, recalling Roger's surmise that they 
were fifth columnists, black hole time-bombs. There wasn't any record of 
mammoth gravitational collapse in the immediate future but the records were 
such a shambles that nothing would have surprised me. 'Tell me, are you 
robots hard-wired to tell the truth?'
           'Categorically,' Marx affirmed stoutly.
	'If you'll forgive me, sir,' Smith added, 'that was a rather pointless 
exchange. You've run up against the paradox of the Cretan Liar, sir. If 
Marx is a liar, how can you trust a word he says?'
           'Quite so.' I cudgelled my brains. Stepping closer, I discerned 
a line of print stamped into their looming hulls, one in English, one in 
Mandarin. Illiterate in either, I asked Roger, 'Is that the statutory 
warrant that these robots are programmed to obey the Three Laws? Answer yes 
or no.'
           'Yes.'
         With a note of resonant ritual, Marx said: 'We avow our adherence 
to the Three Laws of Microprocessors.'
           'First,' said Smith, ` "Thou shalt love mankind with thy whole 
mind and thy whole heart and thy whole soul”.’
           'Second,' cried Marx, ` “Thou shalt love thy neighbor as thyself ".'
           'Third,' finished Smith, clashing its heels together in a crisp 
salute, ` "Thou shalt love thyself”.’
           I was shaken; I'd imagined the behemoths under the control of a 
more stringent algorithm than that. 'It seems rather open to interpretation.'
           'Ethics is like that,' Marx said. 'It's a Gödel problem, like 
the Cretan Liar. Don't fret, though, sir. We're situationalists, but we opt 
from a rather comprehensive metaphysical consensus.'
           That seemed to dispose of the Trojan Horse hypothesis, or at 
least to put it beyond testing.

from: `The Ballad of Bowsprit Bear’s Stead', in EDGES, ed. Ursula K. Le 
Guin & Virginia Kidd, Pocket 1980

Damien Broderick