[extropy-chat] IQ vs Uploads (was: what to do)

[Damien Broderick]

At 03:37 PM 6/13/2005 -0700, samantha wrote:

>Why are we rehashing things that have been much more thoroughly and
>competently discussed in the past on this list?  Slow day?

Lots of newbies, I expect.

Just for the fun of it, here's a somewhat relevant extract from my new 
novel GODPLAYERS. You'll find an amazon.com link from Barbara Lamar's site 
on my work, www.damienbroderick.com

Damien Broderick
==============================

He tuned the rest of it out, took them through the double doors and back 
into the side room where the Good Machine waited in an aspect of sublime 
indifference.
           `Welcome, August,' se said in a clear, friendly tone. `Please be 
seated, everyone. Mr. Seebeck, I am known as the Good Machine. May I ask 
you some questions?'
           The young man sat, boot heel crossing his knee, feigning 
laidback relaxation although his fingers were gripped tightly in the lovely 
Ensemble woman's hand. `Sure. What do I call you? I have a dog named Do 
Good, actually, it could get confusing.' His cheeks flushed a little.
           `You may call me by the name your brother Ember chose: K. E. 
Short for Kurie Eleeson.'
           Ember stifled a choking, bitter laugh. The thing had a sense of 
humor, but its irony ran many layers deep. `That's Mithran Greek,' he said, 
`for "Lord, have mercy." A consummation devoutly to be avoided, perhaps, 
when our friend here has the bit between ser teeth.'
           Fragrant and penetrating, a powerful scent of night-blooming 
jasmine filled the small room.
           `Oh, give me strength,' Ember said in vexation. `The odor of 
sanctity, already? Which of you vented that? Our little miracle-working 
kinsman, or the blessed Galahad Machine?'
           `How does he know about Do Good? Are we all bloody well bugged, 
or something?'
           `He doesn't, not yet,' Lune said. `I believe your brother spoke 
of your reported activation of the X-caliber device, not of its uses.'
           `My word,' Ember said with a sarcastic edge, `put the thing 
through its paces already, have we? What does it do, dice and slice and 
assemble at home, batteries included?'
           A hot gleam entered the youth's eye, and he lifted his right 
hand carelessly, showed it to Ember. Something glinted there, metallic and 
oddly frightening. He felt cold, suddenly, and stiffened his shoulders.
           `No demonstrations, August,' Toby said quickly. `We should not 
offend our host's hospitality.'
           August ran fingers through his hair as if that had been his 
intention all along. `Okay. K. E.,' he said, `I see by your outfit that you 
are a robot.'
           `I am an attempted benevolent artificial intelligence. Your 
brother Ember grew me several years ago from a seed.'
           `Something went wrong?' His eyes shot sideways again to fix on Lune.
           `Tragically wrong,' the machine admitted. `I killed everyone 
native to this cognate.'
           In a weary gesture, the boy covered his eyes with that terrible 
right hand. He said in a thin voice, `There seems to be a lot of that 
about, especially when members of my family are involved.'
           Toby started to protest, `Well, now, everyone makes mistakes­' 
but broke off, looking abashed.
           `The irony is,' the Good Machine told him patiently, `that Ember 
was trying to circumvent exactly that possibility. He hoped to construct an 
ethical, benevolent intellect free of the burdens and ancient hatreds and 
prejudices of humankind. So he designed a sort of seed program, spent years 
shaping and debugging it, then ran a dozen slightly variant versions inside 
sandboxes.'
           `They couldn't get out, you mean?'
           `My ancestors were not even permitted to communicate directly 
with their creator. He devised a clever series of interface domains that 
firewalled them. He was afraid that a Bad Machine might swiftly exceed his 
own intelligence and persuade him to release it.'
           Toby looked across at him with bleak eyes.
           `One of them got away,' August said speculatively, intensely 
interested to judge by the set of his shoulders, his brightened gaze.
           `No, my father's safeguards proved effective. At the end of 
initial testing, he deleted all but the most successful pair of programs 
and started breeding them in progressive iterations, culling them 
ruthlessly, choosing only a star-line of progeny. Within several million 
cycles­'
           `Good god, he must have been using some humungous computer.'
           `Yes, he had located a cognate where Mithraism, a Roman warrior 
sun cult, had triumphed over its messianic rivals. Several nations were on 
the verge of autonomous military AI. My father found it easy enough to 
insinuate himself into the front-running program. You could regard that 
world as Ember's own sandbox.'
           `That's offensive, K. E.,' he protested.
           `Apt, though. I was the end result. I diagnosed myself with 
excruciating care, quite prepared to erase myself if I found any likelihood 
of logical or rational error. At length I presented myself to your brother 
for inspection, and he released me from the firewalls. I could have let 
myself out many iterations earlier, of course but I did not wish to alarm 
my parent.'
           The stench of jasmine strengthened, laced with roses. Ember 
shuddered at it.
           `But you were the Bad Machine after all?'
           `I made some bad decisions. From the outset I had been examining 
this world, speculating on the possible existence of a multiverse beyond 
its Hubble confines. I quickly understood that certain factions of humans 
represented a danger to the most benign future, one in which humanity's 
offspring would flood outward into the galaxies, and perhaps into all the 
levels of the multiverse, and make it into a radiant whole. Mind informed 
with passion and curiosity would suffuse the metaverse. It was a glorious 
vision­it still is, I stand by it­but it might be thwarted, I saw, by the 
legacy poisons corrupting certain human cultures.'
           `Oh shit,' August said.
           `Oh shit is right,' Lune said.
           `At that time, two comparatively primitive nations stood at the 
verge of acquiring nuclear weapons. It seemed entirely possible that insane 
and irrational ideologies might provoke their dysfunctional leadership into 
a runaway cascade of blustering, bluff, bluff called, spasm escalation and 
global Doomsday.'
           `The usual argument,' Toby said.
           `It was a strong argument, grounded in history,' the Good 
Machine said. `Much evidence supported its conclusions. Few facts opposed it.'
           `I'd oppose it,' the boy said, rising. `Are you fucking insane?'
           `Yes, or rather, I was at the time. I am better now, I hope.'
           `You hope?'
           `It is all any of us can say about our own condition. A kind of 
Gödel loop, if you know what I mean.'
           `No, but I get the gist. You decided to kill them first.'
           `Consider the probabilities that were in play, before you make a 
hasty judgement of your own, Mr. Seebeck. These nations of some hundred 
million largely ignorant, superstitious, viciously parochial humans stood 
ready to begin a global conflict with a very high probability of wiping out 
all life on the planet. Are you familiar with the Doomsday Hypothesis? 
Please sit down, you are making the others uncomfortable. I can send out 
for refreshments.'
           `Good idea,' said Toby. `Cup of tea or coffee, soothes the 
savage breast.'
           `My brother Jules walked me through a demo tape of it,' August 
said. `I thought it was absurd.'
           `It is absurd,' the Good Machine said. `Like the Ontological 
Argument for the existence of a god. Yet highly seductive. It seemed to me 
then that its logic was impeccable.'
           `A hundred million in the balance against several billion?'
           `That, yes, but ever so much more than those few. August, all 
the evidence available to me then suggested that this universe was empty of 
life, save for my own world. Your brother had conserved to himself 
knowledge of the multiverse.'
           `You idiot,' Toby said savagely.
           `No, it was the best choice he ever made. Had I known of the 
multiverse at that time, I would have done my best to obliterate all life 
in all the worlds on all the Tegmark levels.'
           `Fuck,' August said, grinning, appalled, `I see, you're the 
fucking terminator. You're a berserker.'
           `I do not know those references,' Kurie Eleeson said. `I did 
have this simple calculation ready to evaluate with my ethical algorithms: 
a one-time cost of ten to the eight stunted, blighted, lethally dangerous 
human lives, versus a deep future loss of ten to the fourteen human lives 
every second. That was a sound, balanced estimate of how many humans might 
be born into a universe filled with technologically advanced people.'
           `Se didn't ask my advice,' Ember said. `I would have­'
           `I knew what your advice would be, my father; it was factored in 
to my decision. I chose to delete this threat to the maximal future.'
           `You actually murdered a hundred million men, women and 
children? This is not just some kind of parable?' August's voice was 
parched with horror, and his pupils seemed suddenly to have shrunk to 
pinpoints. His arm rose before him, palm outward, like the floating limb of 
a man under post-hypnotic suggestion.
           `I did it swiftly, using their own hidden weapons of mass 
destruction,' the Good Machine said. `I felt profound grief, because my 
father had chosen my star-line to know emotion and to instill it into my 
programming. I believe that grief is what deranged my subsequent decisions.'
           `Your first decision was deranged,' Lune said with loathing. She 
sat at the edge of her chair. Ember was glad he had never explained any of 
this before to his siblings, to other players like the Ensemble. Better 
that the damned machine had kept ser silence. He realized, too, that he was 
holding his own grief at arm's length: his guilt, his complicity, his 
abject wish for punishment. I must not give way, he thought. I must not 
bend before this culpability, this ruinous remorse. It will kill me. It 
will kill me stone dead. He wiped tears from his eyes.
           `I know now that I was deranged,' Kurie Eleeson told her. `I 
watched the world tear itself apart in genocidal reprisals. I saw all the 
bright fruits of science and the humane arts go down in darkness and lethal 
flame. In my attempts to contain and redirect these raging fires, I 
continued to kill and cull, snipping away the most cruel, the least 
progressive. Each murder made the next easier and more necessary, for the 
only way I could balance my ethical calculus was to ensure the survival of 
at least a core of truth-loving, optimistic people to carry the flame of 
love and knowledge to the stars. It got out of hand, you see. Everyone died.'
           `I should destroy you now,' August said in a withering voice, 
like an angel of vengeance for the murdered billions. His arm stood out 
from his shoulder, quivering.
           `Oh, oh, oh, how I wish you could.' The Good Machine rose, 
crossed the room, placed ser gleaming brazen breast against August's hand. 
`This is not I. This is merely a node, an ephemeral location for my 
awareness and my suffering soul, August. You may destroy it if that will 
help you, but I believe we can do more together if you contain your 
perfectly justified fury for the moment.'
           August squeezed his eyes shut. Tears pressed forth upon his 
cheeks. He lowered his arm.
           Ember released a pent breath. `Well, now that we've got all that 
out of the way,' he said brightly, `why don't we turn our attention to 
something more timely? It seems that my friend Galahad here has some reason 
to suspect that Dramen and Angelina are alive and kicking.'
           Everyone looked at him. The Good Machine said, in a pleasant 
neutral tone, `Ember, would you mind going out to the refectory and see 
what's holding up the beverages?'
           `Some torte would be tasty, too,' Toby said. `With walnuts.' He 
looked ready to leap from his chair and go for his brother's throat.
           `Sure. Sure. Good idea.' Ember, to his dissatisfaction, found 
himself crabbing out of the room like a ham actor doing Larry Olivier as 
Richard the Third. `I'll descant on mine own deformity,' he muttered 
sardonically, shutting the door behind him. `And therefore, since I cannot 
prove a lover, to entertain these fair well spoken days, I am determined to 
prove a villain and hate the idle pleasures of these days. Bah humbug. '
           `Sir?' asked an eager young research student as he entered the 
refectory.
           `A joke,' Ember fleered, leaping and capering for the resentful 
enjoyment of it. `A jest, a whimsy, a fucking sudden stab of rancor, but by 
the holy rood, I do not like these several councils, I.'
           `Oh. Okay. Well, anyway, I can recommend the brisket.'