[Paleopsych] Wired 8.04: Bill Joy: Why the future doesn't need us.

Premise Checker checker at panix.com
Thu Aug 5 22:29:46 UTC 2004

Bill Joy: Why the future doesn't need us. 
Issue 8.04 - Apr 2000

[This article is very famous and perhaps it is time to read it four years 
later and reconsider it.]

    Our most powerful 21st-century technologies - robotics, genetic
    engineering, and nanotech - are threatening to make humans an
    endangered species.

    From the moment I became involved in the creation of new technologies,
    their ethical dimensions have concerned me, but it was only in the
    autumn of 1998 that I became anxiously aware of how great are the
    dangers facing us in the 21st century. I can date the onset of my
    unease to the day I met Ray Kurzweil, the deservedly famous inventor
    of the first reading machine for the blind and many other amazing

    Ray and I were both speakers at George Gilder's Telecosm conference,
    and I encountered him by chance in the bar of the hotel after both our
    sessions were over. I was sitting with John Searle, a Berkeley
    philosopher who studies consciousness. While we were talking, Ray
    approached and a conversation began, the subject of which haunts me to
    this day.

    I had missed Ray's talk and the subsequent panel that Ray and John had
    been on, and they now picked right up where they'd left off, with Ray
    saying that the rate of improvement of technology was going to
    accelerate and that we were going to become robots or fuse with robots
    or something like that, and John countering that this couldn't happen,
    because the robots couldn't be conscious.

    While I had heard such talk before, I had always felt sentient robots
    were in the realm of science fiction. But now, from someone I
    respected, I was hearing a strong argument that they were a near-term
    possibility. I was taken aback, especially given Ray's proven ability
    to imagine and create the future. I already knew that new technologies
    like genetic engineering and nanotechnology were giving us the power
    to remake the world, but a realistic and imminent scenario for
    intelligent robots surprised me.

    It's easy to get jaded about such breakthroughs. We hear in the news
    almost every day of some kind of technological or scientific advance.
    Yet this was no ordinary prediction. In the hotel bar, Ray gave me a
    partial preprint of his then-forthcoming bookThe Age of Spiritual
    Machines, which outlined a utopia he foresaw - one in which humans
    gained near immortality by becoming one with robotic technology. On
    reading it, my sense of unease only intensified; I felt sure he had to
    be understating the dangers, understating the probability of a bad
    outcome along this path.

    I found myself most troubled by a passage detailing adystopian


    First let us postulate that the computer scientists succeed in
    developing intelligent machines that can do all things better than
    human beings can do them. In that case presumably all work will be
    done by vast, highly organized systems of machines and no human effort
    will be necessary. Either of two cases might occur. The machines might
    be permitted to make all of their own decisions without human
    oversight, or else human control over the machines might be retained.

    If the machines are permitted to make all their own decisions, we
    can't make any conjectures as to the results, because it is impossible
    to guess how such machines might behave. We only point out that the
    fate of the human race would be at the mercy of the machines. It might
    be argued that the human race would never be foolish enough to hand
    over all the power to the machines. But we are suggesting neither that
    the human race would voluntarily turn power over to the machines nor
    that the machines would willfully seize power. What we do suggest is
    that the human race might easily permit itself to drift into a
    position of such dependence on the machines that it would have no
    practical choice but to accept all of the machines' decisions. As
    society and the problems that face it become more and more complex and
    machines become more and more intelligent, people will let machines
    make more of their decisions for them, simply because machine-made
    decisions will bring better results than man-made ones. Eventually a
    stage may be reached at which the decisions necessary to keep the
    system running will be so complex that human beings will be incapable
    of making them intelligently. At that stage the machines will be in
    effective control. People won't be able to just turn the machines off,
    because they will be so dependent on them that turning them off would
    amount to suicide.

    On the other hand it is possible that human control over the machines
    may be retained. In that case the average man may have control over
    certain private machines of his own, such as his car or his personal
    computer, but control over large systems of machines will be in the
    hands of a tiny elite - just as it is today, but with two differences.
    Due to improved techniques the elite will have greater control over
    the masses; and because human work will no longer be necessary the
    masses will be superfluous, a useless burden on the system. If the
    elite is ruthless they may simply decide to exterminate the mass of
    humanity. If they are humane they may use propaganda or other
    psychological or biological techniques to reduce the birth rate until
    the mass of humanity becomes extinct, leaving the world to the elite.
    Or, if the elite consists of soft-hearted liberals, they may decide to
    play the role of good shepherds to the rest of the human race. They
    will see to it that everyone's physical needs are satisfied, that all
    children are raised under psychologically hygienic conditions, that
    everyone has a wholesome hobby to keep him busy, and that anyone who
    may become dissatisfied undergoes "treatment" to cure his "problem."
    Of course, life will be so purposeless that people will have to be
    biologically or psychologically engineered either to remove their need
    for the power process or make them "sublimate" their drive for power
    into some harmless hobby. These engineered human beings may be happy
    in such a society, but they will most certainly not be free. They will
    have been reduced to the status of domestic animals.[15]1

    In the book, you don't discover until you turn the page that the
    author of this passage is Theodore Kaczynski - the Unabomber. I am no
    apologist for Kaczynski. His bombs killed three people during a
    17-year terror campaign and wounded many others. One of his bombs
    gravely injured my friend David Gelernter, one of the most brilliant
    and visionary computer scientists of our time. Like many of my
    colleagues, I felt that I could easily have been the Unabomber's next

    Kaczynski's actions were murderous and, in my view, criminally insane.
    He is clearly a Luddite, but simply saying this does not dismiss his
    argument; as difficult as it is for me to acknowledge, I saw some
    merit in the reasoning in this single passage. I felt compelled to
    confront it.

    Kaczynski's dystopian vision describes unintended consequences, a
    well-known problem with the design and use of technology, and one that
    is clearly related to Murphy's law - "Anything that can go wrong,
    will." (Actually, this is Finagle's law, which in itself shows that
    Finagle was right.) Our overuse of antibiotics has led to what may be
    the biggest such problem so far: the emergence of antibiotic-resistant
    and much more dangerous bacteria. Similar things happened when
    attempts to eliminate malarial mosquitoes using DDT caused them to
    acquire DDT resistance; malarial parasites likewise acquired
    multi-drug-resistant genes.[16]2

    The cause of many such surprises seems clear: The systems involved are
    complex, involving interaction among and feedback between many parts.
    Any changes to such a system will cascade in ways that are difficult
    to predict; this is especially true when human actions are involved.

    I started showing friends the Kaczynski quote fromThe Age of Spiritual
    Machines; I would hand them Kurzweil's book, let them read the quote,
    and then watch their reaction as they discovered who had written it.
    At around the same time, I found Hans Moravec's bookRobot: Mere
    Machine to Transcendent Mind. Moravec is one of the leaders in
    robotics research, and was a founder of the world's largest robotics
    research program, at Carnegie Mellon University.Robot gave me more
    material to try out on my friends - material surprisingly supportive
    of Kaczynski's argument. For example:

    The Short Run (Early 2000s)

    Biological species almost never survive encounters with superior
    competitors. Ten million years ago, South and North America were
    separated by a sunken Panama isthmus. South America, like Australia
    today, was populated by marsupial mammals, including pouched
    equivalents of rats, deers, and tigers. When the isthmus connecting
    North and South America rose, it took only a few thousand years for
    the northern placental species, with slightly more effective
    metabolisms and reproductive and nervous systems, to displace and
    eliminate almost all the southern marsupials.

    In a completely free marketplace, superior robots would surely affect
    humans as North American placentals affected South American marsupials
    (and as humans have affected countless species). Robotic industries
    would compete vigorously among themselves for matter, energy, and
    space, incidentally driving their price beyond human reach. Unable to
    afford the necessities of life, biological humans would be squeezed
    out of existence.

    There is probably some breathing room, because we do not live in a
    completely free marketplace. Government coerces nonmarket behavior,
    especially by collecting taxes. Judiciously applied, governmental
    coercion could support human populations in high style on the fruits
    of robot labor, perhaps for a long while.

    A textbook dystopia - and Moravec is just getting wound up. He goes on
    to discuss how our main job in the 21st century will be "ensuring
    continued cooperation from the robot industries" by passing laws
    decreeing that they be "nice,"[17]3 and to describe how seriously
    dangerous a human can be "once transformed into an unbounded
    superintelligent robot." Moravec's view is that the robots will
    eventually succeed us - that humans clearly face extinction.

    I decided it was time to talk to my friend Danny Hillis. Danny became
    famous as the cofounder of Thinking Machines Corporation, which built
    a very powerful parallel supercomputer. Despite my current job title
    of Chief Scientist at Sun Microsystems, I am more a computer architect
    than a scientist, and I respect Danny's knowledge of the information
    and physical sciences more than that of any other single person I
    know. Danny is also a highly regarded futurist who thinks long-term -
    four years ago he started the Long Now Foundation, which is building a
    clock designed to last 10,000 years, in an attempt to draw attention
    to the pitifully short attention span of our society. (See "[18]Test
    of Time,"Wired 8.03, page 78.)

    So I flew to Los Angeles for the express purpose of having dinner with
    Danny and his wife, Pati. I went through my now-familiar routine,
    trotting out the ideas and passages that I found so disturbing.
    Danny's answer - directed specifically at Kurzweil's scenario of
    humans merging with robots - came swiftly, and quite surprised me. He
    said, simply, that the changes would come gradually, and that we would
    get used to them.

    But I guess I wasn't totally surprised. I had seen a quote from Danny
    in Kurzweil's book in which he said, "I'm as fond of my body as
    anyone, but if I can be 200 with a body of silicon, I'll take it." It
    seemed that he was at peace with this process and its attendant risks,
    while I was not.

    While talking and thinking about Kurzweil, Kaczynski, and Moravec, I
    suddenly remembered a novel I had read almost 20 years ago -The White
    Plague, by Frank Herbert - in which a molecular biologist is driven
    insane by the senseless murder of his family. To seek revenge he
    constructs and disseminates a new and highly contagious plague that
    kills widely but selectively. (We're lucky Kaczynski was a
    mathematician, not a molecular biologist.) I was also reminded of the
    Borg ofStar Trek, a hive of partly biological, partly robotic
    creatures with a strong destructive streak. Borg-like disasters are a
    staple of science fiction, so why hadn't I been more concerned about
    such robotic dystopias earlier? Why weren't other people more
    concerned about these nightmarish scenarios?

    Part of the answer certainly lies in our attitude toward the new - in
    our bias toward instant familiarity and unquestioning acceptance.
    Accustomed to living with almost routine scientific breakthroughs, we
    have yet to come to terms with the fact that the most compelling
    21st-century technologies - robotics, genetic engineering, and
    nanotechnology - pose a different threat than the technologies that
    have come before. Specifically, robots, engineered organisms, and
    nanobots share a dangerous amplifying factor: They can self-replicate.
    A bomb is blown up only once - but one bot can become many, and
    quickly get out of control.

    Much of my work over the past 25 years has been on computer
    networking, where the sending and receiving of messages creates the
    opportunity for out-of-control replication. But while replication in a
    computer or a computer network can be a nuisance, at worst it disables
    a machine or takes down a network or network service. Uncontrolled
    self-replication in these newer technologies runs a much greater risk:
    a risk of substantial damage in the physical world.

    Each of these technologies also offers untold promise: The vision of
    near immortality that Kurzweil sees in his robot dreams drives us
    forward; genetic engineering may soon provide treatments, if not
    outright cures, for most diseases; and nanotechnology and nanomedicine
    can address yet more ills. Together they could significantly extend
    our average life span and improve the quality of our lives. Yet, with
    each of these technologies, a sequence of small, individually sensible
    advances leads to an accumulation of great power and, concomitantly,
    great danger.

    What was different in the 20th century? Certainly, the technologies
    underlying the weapons of mass destruction (WMD) - nuclear,
    biological, and chemical (NBC) - were powerful, and the weapons an
    enormous threat. But building nuclear weapons required, at least for a
    time, access to both rare - indeed, effectively unavailable - raw
    materials and highly protected information; biological and chemical
    weapons programs also tended to require large-scale activities.

    The 21st-century technologies - genetics, nanotechnology, and robotics
    (GNR) - are so powerful that they can spawn whole new classes of
    accidents and abuses. Most dangerously, for the first time, these
    accidents and abuses are widely within the reach of individuals or
    small groups. They will not require large facilities or rare raw
    materials. Knowledge alone will enable the use of them.

    Thus we have the possibility not just of weapons of mass destruction
    but of knowledge-enabled mass destruction (KMD), this destructiveness
    hugely amplified by the power of self-replication.

    I think it is no exaggeration to say we are on the cusp of the further
    perfection of extreme evil, an evil whose possibility spreads well
    beyond that which weapons of mass destruction bequeathed to the
    nation-states, on to a surprising and terrible empowerment of extreme

    Nothing about the way I got involved with computers suggested to me
    that I was going to be facing these kinds of issues.

    My life has been driven by a deep need to ask questions and find
    answers. When I was 3, I was already reading, so my father took me to
    the elementary school, where I sat on the principal's lap and read him
    a story. I started school early, later skipped a grade, and escaped
    into books - I was incredibly motivated to learn. I asked lots of
    questions, often driving adults to distraction.

    As a teenager I was very interested in science and technology. I
    wanted to be a ham radio operator but didn't have the money to buy the
    equipment. Ham radio was the Internet of its time: very addictive, and
    quite solitary. Money issues aside, my mother put her foot down - I
    was not to be a ham; I was antisocial enough already.

    I may not have had many close friends, but I was awash in ideas. By
    high school, I had discovered the great science fiction writers. I
    remember especially Heinlein'sHave Spacesuit Will Travel and Asimov's
    I, Robot, with its Three Laws of Robotics. I was enchanted by the
    descriptions of space travel, and wanted to have a telescope to look
    at the stars; since I had no money to buy or make one, I checked books
    on telescope-making out of the library and read about making them
    instead. I soared in my imagination.

    Thursday nights my parents went bowling, and we kids stayed home
    alone. It was the night of Gene Roddenberry's original Star Trek, and
    the program made a big impression on me. I came to accept its notion
    that humans had a future in space, Western-style, with big heroes and
    adventures. Roddenberry's vision of the centuries to come was one with
    strong moral values, embodied in codes like the Prime Directive: to
    not interfere in the development of less technologically advanced
    civilizations. This had an incredible appeal to me; ethical humans,
    not robots, dominated this future, and I took Roddenberry's dream as
    part of my own.

    I excelled in mathematics in high school, and when I went to the
    University of Michigan as an undergraduate engineering student I took
    the advanced curriculum of the mathematics majors. Solving math
    problems was an exciting challenge, but when I discovered computers I
    found something much more interesting: a machine into which you could
    put a program that attempted to solve a problem, after which the
    machine quickly checked the solution. The computer had a clear notion
    of correct and incorrect, true and false. Were my ideas correct? The
    machine could tell me. This was very seductive.

    I was lucky enough to get a job programming early supercomputers and
    discovered the amazing power of large machines to numerically simulate
    advanced designs. When I went to graduate school at UC Berkeley in the
    mid-1970s, I started staying up late, often all night, inventing new
    worlds inside the machines. Solving problems. Writing the code that
    argued so strongly to be written.

    InThe Agony and the Ecstasy, Irving Stone's biographical novel of
    Michelangelo, Stone described vividly how Michelangelo released the
    statues from the stone, "breaking the marble spell," carving from the
    images in his mind.[19]4 In my most ecstatic moments, the software in
    the computer emerged in the same way. Once I had imagined it in my
    mind I felt that it was already there in the machine, waiting to be
    released. Staying up all night seemed a small price to pay to free it
    - to give the ideas concrete form.

    After a few years at Berkeley I started to send out some of the
    software I had written - an instructional Pascal system, Unix
    utilities, and a text editor called vi (which is still, to my
    surprise, widely used more than 20 years later) - to others who had
    similar small PDP-11 and VAX minicomputers. These adventures in
    software eventually turned into the Berkeley version of the Unix
    operating system, which became a personal "success disaster" - so many
    people wanted it that I never finished my PhD. Instead I got a job
    working for Darpa putting Berkeley Unix on the Internet and fixing it
    to be reliable and to run large research applications well. This was
    all great fun and very rewarding. And, frankly, I saw no robots here,
    or anywhere near.

    Still, by the early 1980s, I was drowning. The Unix releases were very
    successful, and my little project of one soon had money and some
    staff, but the problem at Berkeley was always office space rather than
    money - there wasn't room for the help the project needed, so when the
    other founders of Sun Microsystems showed up I jumped at the chance to
    join them. At Sun, the long hours continued into the early days of
    workstations and personal computers, and I have enjoyed participating
    in the creation of advanced microprocessor technologies and Internet
    technologies such as Java and Jini.

    From all this, I trust it is clear that I am not a Luddite. I have
    always, rather, had a strong belief in the value of the scientific
    search for truth and in the ability of great engineering to bring
    material progress. The Industrial Revolution has immeasurably improved
    everyone's life over the last couple hundred years, and I always
    expected my career to involve the building of worthwhile solutions to
    real problems, one problem at a time.

    I have not been disappointed. My work has had more impact than I had
    ever hoped for and has been more widely used than I could have
    reasonably expected. I have spent the last 20 years still trying to
    figure out how to make computers as reliable as I want them to be
    (they are not nearly there yet) and how to make them simple to use (a
    goal that has met with even less relative success). Despite some
    progress, the problems that remain seem even more daunting.

    But while I was aware of the moral dilemmas surrounding technology's
    consequences in fields like weapons research, I did not expect that I
    would confront such issues in my own field, or at least not so soon.

    Perhaps it is always hard to see the bigger impact while you are in
    the vortex of a change. Failing to understand the consequences of our
    inventions while we are in the rapture of discovery and innovation
    seems to be a common fault of scientists and technologists; we have
    long been driven by the overarching desire to know that is the nature
    of science's quest, not stopping to notice that the progress to newer
    and more powerful technologies can take on a life of its own.

    I have long realized that the big advances in information technology
    come not from the work of computer scientists, computer architects, or
    electrical engineers, but from that of physical scientists. The
    physicists Stephen Wolfram and Brosl Hasslacher introduced me, in the
    early 1980s, to chaos theory and nonlinear systems. In the 1990s, I
    learned about complex systems from conversations with Danny Hillis,
    the biologist Stuart Kauffman, the Nobel-laureate physicist Murray
    Gell-Mann, and others. Most recently, Hasslacher and the electrical
    engineer and device physicist Mark Reed have been giving me insight
    into the incredible possibilities of molecular electronics.

    In my own work, as codesigner of three microprocessor architectures -
    SPARC, picoJava, and MAJC - and as the designer of several
    implementations thereof, I've been afforded a deep and firsthand
    acquaintance with Moore's law. For decades, Moore's law has correctly
    predicted the exponential rate of improvement of semiconductor
    technology. Until last year I believed that the rate of advances
    predicted by Moore's law might continue only until roughly 2010, when
    some physical limits would begin to be reached. It was not obvious to
    me that a new technology would arrive in time to keep performance
    advancing smoothly.

    But because of the recent rapid and radical progress in molecular
    electronics - where individual atoms and molecules replace
    lithographically drawn transistors - and related nanoscale
    technologies, we should be able to meet or exceed the Moore's law rate
    of progress for another 30 years. By 2030, we are likely to be able to
    build machines, in quantity, a million times as powerful as the
    personal computers of today - sufficient to implement the dreams of
    Kurzweil and Moravec.

    As this enormous computing power is combined with the manipulative
    advances of the physical sciences and the new, deep understandings in
    genetics, enormous transformative power is being unleashed. These
    combinations open up the opportunity to completely redesign the world,
    for better or worse: The replicating and evolving processes that have
    been confined to the natural world are about to become realms of human

    In designing software and microprocessors, I have never had the
    feeling that I was designing an intelligent machine. The software and
    hardware is so fragile and the capabilities of the machine to "think"
    so clearly absent that, even as a possibility, this has always seemed
    very far in the future.

    But now, with the prospect of human-level computing power in about 30
    years, a new idea suggests itself: that I may be working to create
    tools which will enable the construction of the technology that may
    replace our species. How do I feel about this? Very uncomfortable.
    Having struggled my entire career to build reliable software systems,
    it seems to me more than likely that this future will not work out as
    well as some people may imagine. My personal experience suggests we
    tend to overestimate our design abilities.

    Given the incredible power of these new technologies, shouldn't we be
    asking how we can best coexist with them? And if our own extinction is
    a likely, or even possible, outcome of our technological development,
    shouldn't we proceed with great caution?

    The dream of robotics is, first, that intelligent machines can do our
    work for us, allowing us lives of leisure, restoring us to Eden. Yet
    in his history of such ideas,Darwin Among the Machines, George Dyson
    warns: "In the game of life and evolution there are three players at
    the table: human beings, nature, and machines. I am firmly on the side
    of nature. But nature, I suspect, is on the side of the machines." As
    we have seen, Moravec agrees, believing we may well not survive the
    encounter with the superior robot species.

    How soon could such an intelligent robot be built? The coming advances
    in computing power seem to make it possible by 2030. And once an
    intelligent robot exists, it is only a small step to a robot species -
    to an intelligent robot that can make evolved copies of itself.

    A second dream of robotics is that we will gradually replace ourselves
    with our robotic technology, achieving near immortality by downloading
    our consciousnesses; it is this process that Danny Hillis thinks we
    will gradually get used to and that Ray Kurzweil elegantly details
    inThe Age of Spiritual Machines. (We are beginning to see intimations
    of this in the implantation of computer devices into the human body,
    as illustrated on the[20]cover ofWired 8.02.)

    But if we are downloaded into our technology, what are the chances
    that we will thereafter be ourselves or even human? It seems to me far
    more likely that a robotic existence would not be like a human one in
    any sense that we understand, that the robots would in no sense be our
    children, that on this path our humanity may well be lost.

    Genetic engineering promises to revolutionize agriculture by
    increasing crop yields while reducing the use of pesticides; to create
    tens of thousands of novel species of bacteria, plants, viruses, and
    animals; to replace reproduction, or supplement it, with cloning; to
    create cures for many diseases, increasing our life span and our
    quality of life; and much, much more. We now know with certainty that
    these profound changes in the biological sciences are imminent and
    will challenge all our notions of what life is.

    Technologies such as human cloning have in particular raised our
    awareness of the profound ethical and moral issues we face. If, for
    example, we were to reengineer ourselves into several separate and
    unequal species using the power of genetic engineering, then we would
    threaten the notion of equality that is the very cornerstone of our

    Given the incredible power of genetic engineering, it's no surprise
    that there are significant safety issues in its use. My friend Amory
    Lovins recently cowrote, along with Hunter Lovins, an editorial that
    provides an ecological view of some of these dangers. Among their
    concerns: that "the new botany aligns the development of plants with
    their economic, not evolutionary, success." (See "[21]A Tale of Two
    Botanies," page 247.) Amory's long career has been focused on energy
    and resource efficiency by taking a whole-system view of human-made
    systems; such a whole-system view often finds simple, smart solutions
    to otherwise seemingly difficult problems, and is usefully applied
    here as well.

    After reading the Lovins' editorial, I saw an op-ed by Gregg
    Easterbrook inThe New York Times (November 19, 1999) about genetically
    engineered crops, under the headline: "Food for the Future: Someday,
    rice will have built-in vitamin A. Unless the Luddites win."

    Are Amory and Hunter Lovins Luddites? Certainly not. I believe we all
    would agree that golden rice, with its built-in vitamin A, is probably
    a good thing, if developed with proper care and respect for the likely
    dangers in moving genes across species boundaries.

    Awareness of the dangers inherent in genetic engineering is beginning
    to grow, as reflected in the Lovins' editorial. The general public is
    aware of, and uneasy about, genetically modified foods, and seems to
    be rejecting the notion that such foods should be permitted to be

    But genetic engineering technology is already very far along. As the
    Lovins note, the USDA has already approved about 50 genetically
    engineered crops for unlimited release; more than half of the world's
    soybeans and a third of its corn now contain genes spliced in from
    other forms of life.

    While there are many important issues here, my own major concern with
    genetic engineering is narrower: that it gives the power - whether
    militarily, accidentally, or in a deliberate terrorist act - to create
    a White Plague.

    The many wonders of nanotechnology were first imagined by the
    Nobel-laureate physicist Richard Feynman in a speech he gave in 1959,
    subsequently published under the title "There's Plenty of Room at the
    Bottom." The book that made a big impression on me, in the mid-'80s,
    was Eric Drexler'sEngines of Creation, in which he described
    beautifully how manipulation of matter at the atomic level could
    create a utopian future of abundance, where just about everything
    could be made cheaply, and almost any imaginable disease or physical
    problem could be solved using nanotechnology and artificial

    A subsequent book,Unbounding the Future: The Nanotechnology
    Revolution, which Drexler cowrote, imagines some of the changes that
    might take place in a world where we had molecular-level "assemblers."
    Assemblers could make possible incredibly low-cost solar power, cures
    for cancer and the common cold by augmentation of the human immune
    system, essentially complete cleanup of the environment, incredibly
    inexpensive pocket supercomputers - in fact, any product would be
    manufacturable by assemblers at a cost no greater than that of wood -
    spaceflight more accessible than transoceanic travel today, and
    restoration of extinct species.

    I remember feeling good about nanotechnology after readingEngines of
    Creation. As a technologist, it gave me a sense of calm - that is,
    nanotechnology showed us that incredible progress was possible, and
    indeed perhaps inevitable. If nanotechnology was our future, then I
    didn't feel pressed to solve so many problems in the present. I would
    get to Drexler's utopian future in due time; I might as well enjoy
    life more in the here and now. It didn't make sense, given his vision,
    to stay up all night, all the time.

    Drexler's vision also led to a lot of good fun. I would occasionally
    get to describe the wonders of nanotechnology to others who had not
    heard of it. After teasing them with all the things Drexler described
    I would give a homework assignment of my own: "Use nanotechnology to
    create a vampire; for extra credit create an antidote."

    With these wonders came clear dangers, of which I was acutely aware.
    As I said at a nanotechnology conference in 1989, "We can't simply do
    our science and not worry about these ethical issues."[22]5 But my
    subsequent conversations with physicists convinced me that
    nanotechnology might not even work - or, at least, it wouldn't work
    anytime soon. Shortly thereafter I moved to Colorado, to a skunk works
    I had set up, and the focus of my work shifted to software for the
    Internet, specifically on ideas that became Java and Jini.

    Then, last summer, Brosl Hasslacher told me that nanoscale molecular
    electronics was now practical. This wasnew news, at least to me, and I
    think to many people - and it radically changed my opinion about
    nanotechnology. It sent me back toEngines of Creation. Rereading
    Drexler's work after more than 10 years, I was dismayed to realize how
    little I had remembered of its lengthy section called "Dangers and
    Hopes," including a discussion of how nanotechnologies can become
    "engines of destruction." Indeed, in my rereading of this cautionary
    material today, I am struck by how naive some of Drexler's safeguard
    proposals seem, and how much greater I judge the dangers to be now
    than even he seemed to then. (Having anticipated and described many
    technical and political problems with nanotechnology, Drexler started
    the Foresight Institute in the late 1980s "to help prepare society for
    anticipated advanced technologies" - most important, nanotechnology.)

    The enabling breakthrough to assemblers seems quite likely within the
    next 20 years. Molecular electronics - the new subfield of
    nanotechnology where individual molecules are circuit elements -
    should mature quickly and become enormously lucrative within this
    decade, causing a large incremental investment in all

    Unfortunately, as with nuclear technology, it is far easier to create
    destructive uses for nanotechnology than constructive ones.
    Nanotechnology has clear military and terrorist uses, and you need not
    be suicidal to release a massively destructive nanotechnological
    device - such devices can be built to be selectively destructive,
    affecting, for example, only a certain geographical area or a group of
    people who are genetically distinct.

    An immediate consequence of the Faustian bargain in obtaining the
    great power of nanotechnology is that we run a grave risk - the risk
    that we might destroy the biosphere on which all life depends.

    As Drexler explained:

    "Plants" with "leaves" no more efficient than today's solar cells
    could out-compete real plants, crowding the biosphere with an inedible
    foliage. Tough omnivorous "bacteria" could out-compete real bacteria:
    They could spread like blowing pollen, replicate swiftly, and reduce
    the biosphere to dust in a matter of days. Dangerous replicators could
    easily be too tough, small, and rapidly spreading to stop - at least
    if we make no preparation. We have trouble enough controlling viruses
    and fruit flies.

    Among the cognoscenti of nanotechnology, this threat has become known
    as the "gray goo problem." Though masses of uncontrolled replicators
    need not be gray or gooey, the term "gray goo" emphasizes that
    replicators able to obliterate life might be less inspiring than a
    single species of crabgrass. They might be superior in an evolutionary
    sense, but this need not make them valuable.

    The gray goo threat makes one thing perfectly clear: We cannot afford
    certain kinds of accidents with replicating assemblers.

    Gray goo would surely be a depressing ending to our human adventure on
    Earth, far worse than mere fire or ice, and one that could stem from a
    simple laboratory accident.[23]6 Oops.

    It is most of all the power of destructive self-replication in
    genetics, nanotechnology, and robotics (GNR) that should give us
    pause. Self-replication is the modus operandi of genetic engineering,
    which uses the machinery of the cell to replicate its designs, and the
    prime danger underlying gray goo in nanotechnology. Stories of
    run-amok robots like the Borg, replicating or mutating to escape from
    the ethical constraints imposed on them by their creators, are well
    established in our science fiction books and movies. It is even
    possible that self-replication may be more fundamental than we
    thought, and hence harder - or even impossible - to control. A recent
    article by Stuart Kauffman inNature titled "Self-Replication: Even
    Peptides Do It" discusses the discovery that a 32-amino-acid peptide
    can "autocatalyse its own synthesis." We don't know how widespread
    this ability is, but Kauffman notes that it may hint at "a route to
    self-reproducing molecular systems on a basis far wider than
    Watson-Crick base-pairing."[24]7

    In truth, we have had in hand for years clear warnings of the dangers
    inherent in widespread knowledge of GNR technologies - of the
    possibility of knowledge alone enabling mass destruction. But these
    warnings haven't been widely publicized; the public discussions have
    been clearly inadequate. There is no profit in publicizing the

    The nuclear, biological, and chemical (NBC) technologies used in
    20th-century weapons of mass destruction were and are largely
    military, developed in government laboratories. In sharp contrast, the
    21st-century GNR technologies have clear commercial uses and are being
    developed almost exclusively by corporate enterprises. In this age of
    triumphant commercialism, technology - with science as its handmaiden
    - is delivering a series of almost magical inventions that are the
    most phenomenally lucrative ever seen. We are aggressively pursuing
    the promises of these new technologies within the now-unchallenged
    system of global capitalism and its manifold financial incentives and
    competitive pressures.

    This is the first moment in the history of our planet when any
    species, by its own voluntary actions, has become a danger to itself -
    as well as to vast numbers of others.

    It might be a familiar progression, transpiring on many worlds - a
    planet, newly formed, placidly revolves around its star; life slowly
    forms; a kaleidoscopic procession of creatures evolves; intelligence
    emerges which, at least up to a point, confers enormous survival
    value; and then technology is invented. It dawns on them that there
    are such things as laws of Nature, that these laws can be revealed by
    experiment, and that knowledge of these laws can be made both to save
    and to take lives, both on unprecedented scales. Science, they
    recognize, grants immense powers. In a flash, they create
    world-altering contrivances. Some planetary civilizations see their
    way through, place limits on what may and what must not be done, and
    safely pass through the time of perils. Others, not so lucky or so
    prudent, perish.

    That is Carl Sagan, writing in 1994, inPale Blue Dot, a book
    describing his vision of the human future in space. I am only now
    realizing how deep his insight was, and how sorely I miss, and will
    miss, his voice. For all its eloquence, Sagan's contribution was not
    least that of simple common sense - an attribute that, along with
    humility, many of the leading advocates of the 21st-century
    technologies seem to lack.

    I remember from my childhood that my grandmother was strongly against
    the overuse of antibiotics. She had worked since before the first
    World War as a nurse and had a commonsense attitude that taking
    antibiotics, unless they were absolutely necessary, was bad for you.

    It is not that she was an enemy of progress. She saw much progress in
    an almost 70-year nursing career; my grandfather, a diabetic,
    benefited greatly from the improved treatments that became available
    in his lifetime. But she, like many levelheaded people, would probably
    think it greatly arrogant for us, now, to be designing a robotic
    "replacement species," when we obviously have so much trouble making
    relatively simple things work, and so much trouble managing - or even
    understanding - ourselves.

    I realize now that she had an awareness of the nature of the order of
    life, and of the necessity of living with and respecting that order.
    With this respect comes a necessary humility that we, with our
    early-21st-century chutzpah, lack at our peril. The commonsense view,
    grounded in this respect, is often right, in advance of the scientific
    evidence. The clear fragility and inefficiencies of the human-made
    systems we have built should give us all pause; the fragility of the
    systems I have worked on certainly humbles me.

    We should have learned a lesson from the making of the first atomic
    bomb and the resulting arms race. We didn't do well then, and the
    parallels to our current situation are troubling.

    The effort to build the first atomic bomb was led by the brilliant
    physicist J. Robert Oppenheimer. Oppenheimer was not naturally
    interested in politics but became painfully aware of what he perceived
    as the grave threat to Western civilization from the Third Reich, a
    threat surely grave because of the possibility that Hitler might
    obtain nuclear weapons. Energized by this concern, he brought his
    strong intellect, passion for physics, and charismatic leadership
    skills to Los Alamos and led a rapid and successful effort by an
    incredible collection of great minds to quickly invent the bomb.

    What is striking is how this effort continued so naturally after the
    initial impetus was removed. In a meeting shortly after V-E Day with
    some physicists who felt that perhaps the effort should stop,
    Oppenheimer argued to continue. His stated reason seems a bit strange:
    not because of the fear of large casualties from an invasion of Japan,
    but because the United Nations, which was soon to be formed, should
    have foreknowledge of atomic weapons. A more likely reason the project
    continued is the momentum that had built up - the first atomic test,
    Trinity, was nearly at hand.

    We know that in preparing this first atomic test the physicists
    proceeded despite a large number of possible dangers. They were
    initially worried, based on a calculation by Edward Teller, that an
    atomic explosion might set fire to the atmosphere. A revised
    calculation reduced the danger of destroying the world to a
    three-in-a-million chance. (Teller says he was later able to dismiss
    the prospect of atmospheric ignition entirely.) Oppenheimer, though,
    was sufficiently concerned about the result of Trinity that he
    arranged for a possible evacuation of the southwest part of the state
    of New Mexico. And, of course, there was the clear danger of starting
    a nuclear arms race.

    Within a month of that first, successful test, two atomic bombs
    destroyed Hiroshima and Nagasaki. Some scientists had suggested that
    the bomb simply be demonstrated, rather than dropped on Japanese
    cities - saying that this would greatly improve the chances for arms
    control after the war - but to no avail. With the tragedy of Pearl
    Harbor still fresh in Americans' minds, it would have been very
    difficult for President Truman to order a demonstration of the weapons
    rather than use them as he did - the desire to quickly end the war and
    save the lives that would have been lost in any invasion of Japan was
    very strong. Yet the overriding truth was probably very simple: As the
    physicist Freeman Dyson later said, "The reason that it was dropped
    was just that nobody had the courage or the foresight to say no."

    It's important to realize how shocked the physicists were in the
    aftermath of the bombing of Hiroshima, on August 6, 1945. They
    describe a series of waves of emotion: first, a sense of fulfillment
    that the bomb worked, then horror at all the people that had been
    killed, and then a convincing feeling that on no account should
    another bomb be dropped. Yet of course another bomb was dropped, on
    Nagasaki, only three days after the bombing of Hiroshima.

    In November 1945, three months after the atomic bombings, Oppenheimer
    stood firmly behind the scientific attitude, saying, "It is not
    possible to be a scientist unless you believe that the knowledge of
    the world, and the power which this gives, is a thing which is of
    intrinsic value to humanity, and that you are using it to help in the
    spread of knowledge and are willing to take the consequences."

    Oppenheimer went on to work, with others, on the Acheson-Lilienthal
    report, which, as Richard Rhodes says in his recent bookVisions of
    Technology, "found a way to prevent a clandestine nuclear arms race
    without resorting to armed world government"; their suggestion was a
    form of relinquishment of nuclear weapons work by nation-states to an
    international agency.

    This proposal led to the Baruch Plan, which was submitted to the
    United Nations in June 1946 but never adopted (perhaps because, as
    Rhodes suggests, Bernard Baruch had "insisted on burdening the plan
    with conventional sanctions," thereby inevitably dooming it, even
    though it would "almost certainly have been rejected by Stalinist
    Russia anyway"). Other efforts to promote sensible steps toward
    internationalizing nuclear power to prevent an arms race ran afoul
    either of US politics and internal distrust, or distrust by the
    Soviets. The opportunity to avoid the arms race was lost, and very

    Two years later, in 1948, Oppenheimer seemed to have reached another
    stage in his thinking, saying, "In some sort of crude sense which no
    vulgarity, no humor, no overstatement can quite extinguish, the
    physicists have known sin; and this is a knowledge they cannot lose."

    In 1949, the Soviets exploded an atom bomb. By 1955, both the US and
    the Soviet Union had tested hydrogen bombs suitable for delivery by
    aircraft. And so the nuclear arms race began.

    Nearly 20 years ago, in the documentaryThe Day After Trinity, Freeman
    Dyson summarized the scientific attitudes that brought us to the
    nuclear precipice:

    "I have felt it myself. The glitter of nuclear weapons. It is
    irresistible if you come to them as a scientist. To feel it's there in
    your hands, to release this energy that fuels the stars, to let it do
    your bidding. To perform these miracles, to lift a million tons of
    rock into the sky. It is something that gives people an illusion of
    illimitable power, and it is, in some ways, responsible for all our
    troubles - this, what you might call technical arrogance, that
    overcomes people when they see what they can do with their

    Now, as then, we are creators of new technologies and stars of the
    imagined future, driven - this time by great financial rewards and
    global competition - despite the clear dangers, hardly evaluating what
    it may be like to try to live in a world that is the realistic outcome
    of what we are creating and imagining.

    In 1947,The Bulletin of the Atomic Scientists began putting a Doomsday
    Clock on its cover. For more than 50 years, it has shown an estimate
    of the relative nuclear danger we have faced, reflecting the changing
    international conditions. The hands on the clock have moved 15 times
    and today, standing at nine minutes to midnight, reflect continuing
    and real danger from nuclear weapons. The recent addition of India and
    Pakistan to the list of nuclear powers has increased the threat of
    failure of the nonproliferation goal, and this danger was reflected by
    moving the hands closer to midnight in 1998.

    In our time, how much danger do we face, not just from nuclear
    weapons, but from all of these technologies? How high are the
    extinction risks?

    The philosopher John Leslie has studied this question and concluded
    that the risk of human extinction is at least 30 percent,[26]9 while
    Ray Kurzweil believes we have "a better than even chance of making it
    through," with the caveat that he has "always been accused of being an
    optimist." Not only are these estimates not encouraging, but they do
    not include the probability of many horrid outcomes that lie short of

    Faced with such assessments, some serious people are already
    suggesting that we simply move beyond Earth as quickly as possible. We
    would colonize the galaxy using von Neumann probes, which hop from
    star system to star system, replicating as they go. This step will
    almost certainly be necessary 5 billion years from now (or sooner if
    our solar system is disastrously impacted by the impending collision
    of our galaxy with the Andromeda galaxy within the next 3 billion
    years), but if we take Kurzweil and Moravec at their word it might be
    necessary by the middle of this century.

    What are the moral implications here? If we must move beyond Earth
    this quickly in order for the species to survive, who accepts the
    responsibility for the fate of those (most of us, after all) who are
    left behind? And even if we scatter to the stars, isn't it likely that
    we may take our problems with us or find, later, that they have
    followed us? The fate of our species on Earth and our fate in the
    galaxy seem inextricably linked.

    Another idea is to erect a series of shields to defend against each of
    the dangerous technologies. The Strategic Defense Initiative, proposed
    by the Reagan administration, was an attempt to design such a shield
    against the threat of a nuclear attack from the Soviet Union. But as
    Arthur C. Clarke, who was privy to discussions about the project,
    observed: "Though it might be possible, at vast expense, to construct
    local defense systems that would 'only' let through a few percent of
    ballistic missiles, the much touted idea of a national umbrella was
    nonsense. Luis Alvarez, perhaps the greatest experimental physicist of
    this century, remarked to me that the advocates of such schemes were
    'very bright guys with no common sense.'"

    Clarke continued: "Looking into my often cloudy crystal ball, I
    suspect that a total defense might indeed be possible in a century or
    so. But the technology involved would produce, as a by-product,
    weapons so terrible that no one would bother with anything as
    primitive as ballistic missiles." [27]10

    InEngines of Creation, Eric Drexler proposed that we build an active
    nanotechnological shield - a form of immune system for the biosphere -
    to defend against dangerous replicators of all kinds that might escape
    from laboratories or otherwise be maliciously created. But the shield
    he proposed would itself be extremely dangerous - nothing could
    prevent it from developing autoimmune problems and attacking the
    biosphere itself. [28]11

    Similar difficulties apply to the construction of shields against
    robotics and genetic engineering. These technologies are too powerful
    to be shielded against in the time frame of interest; even if it were
    possible to implement defensive shields, the side effects of their
    development would be at least as dangerous as the technologies we are
    trying to protect against.

    These possibilities are all thus either undesirable or unachievable or
    both. The only realistic alternative I see is relinquishment: to limit
    development of the technologies that are too dangerous, by limiting
    our pursuit of certain kinds of knowledge.

    Yes, I know, knowledge is good, as is the search for new truths. We
    have been seeking knowledge since ancient times. Aristotle opened his
    Metaphysics with the simple statement: "All men by nature desire to
    know." We have, as a bedrock value in our society, long agreed on the
    value of open access to information, and recognize the problems that
    arise with attempts to restrict access to and development of
    knowledge. In recent times, we have come to revere scientific

    But despite the strong historical precedents, if open access to and
    unlimited development of knowledge henceforth puts us all in clear
    danger of extinction, then common sense demands that we reexamine even
    these basic, long-held beliefs.

    It was Nietzsche who warned us, at the end of the 19th century, not
    only that God is dead but that "faith in science, which after all
    exists undeniably, cannot owe its origin to a calculus of utility; it
    must have originated in spite of the fact that the disutility and
    dangerousness of the 'will to truth,' of 'truth at any price' is
    proved to it constantly." It is this further danger that we now fully
    face - the consequences of our truth-seeking. The truth that science
    seeks can certainly be considered a dangerous substitute for God if it
    is likely to lead to our extinction.

    If we could agree, as a species, what we wanted, where we were headed,
    and why, then we would make our future much less dangerous - then we
    might understand what we can and should relinquish. Otherwise, we can
    easily imagine an arms race developing over GNR technologies, as it
    did with the NBC technologies in the 20th century. This is perhaps the
    greatest risk, for once such a race begins, it's very hard to end it.
    This time - unlike during the Manhattan Project - we aren't in a war,
    facing an implacable enemy that is threatening our civilization; we
    are driven, instead, by our habits, our desires, our economic system,
    and our competitive need to know.

    I believe that we all wish our course could be determined by our
    collective values, ethics, and morals. If we had gained more
    collective wisdom over the past few thousand years, then a dialogue to
    this end would be more practical, and the incredible powers we are
    about to unleash would not be nearly so troubling.

    One would think we might be driven to such a dialogue by our instinct
    for self-preservation. Individuals clearly have this desire, yet as a
    species our behavior seems to be not in our favor. In dealing with the
    nuclear threat, we often spoke dishonestly to ourselves and to each
    other, thereby greatly increasing the risks. Whether this was
    politically motivated, or because we chose not to think ahead, or
    because when faced with such grave threats we acted irrationally out
    of fear, I do not know, but it does not bode well.

    The new Pandora's boxes of genetics, nanotechnology, and robotics are
    almost open, yet we seem hardly to have noticed. Ideas can't be put
    back in a box; unlike uranium or plutonium, they don't need to be
    mined and refined, and they can be freely copied. Once they are out,
    they are out. Churchill remarked, in a famous left-handed compliment,
    that the American people and their leaders "invariably do the right
    thing, after they have examined every other alternative." In this
    case, however, we must act more presciently, as to do the right thing
    only at last may be to lose the chance to do it at all.

    As Thoreau said, "We do not ride on the railroad; it rides upon us";
    and this is what we must fight, in our time. The question is, indeed,
    Which is to be master? Will we survive our technologies?

    We are being propelled into this new century with no plan, no control,
    no brakes. Have we already gone too far down the path to alter course?
    I don't believe so, but we aren't trying yet, and the last chance to
    assert control - the fail-safe point - is rapidly approaching. We have
    our first pet robots, as well as commercially available genetic
    engineering techniques, and our nanoscale techniques are advancing
    rapidly. While the development of these technologies proceeds through
    a number of steps, it isn't necessarily the case - as happened in the
    Manhattan Project and the Trinity test - that the last step in proving
    a technology is large and hard. The breakthrough to wild
    self-replication in robotics, genetic engineering, or nanotechnology
    could come suddenly, reprising the surprise we felt when we learned of
    the cloning of a mammal.

    And yet I believe we do have a strong and solid basis for hope. Our
    attempts to deal with weapons of mass destruction in the last century
    provide a shining example of relinquishment for us to consider: the
    unilateral US abandonment, without preconditions, of the development
    of biological weapons. This relinquishment stemmed from the
    realization that while it would take an enormous effort to create
    these terrible weapons, they could from then on easily be duplicated
    and fall into the hands of rogue nations or terrorist groups.

    The clear conclusion was that we would create additional threats to
    ourselves by pursuing these weapons, and that we would be more secure
    if we did not pursue them. We have embodied our relinquishment of
    biological and chemical weapons in the 1972 Biological Weapons
    Convention (BWC) and the 1993 Chemical Weapons Convention (CWC).[29]12

    As for the continuing sizable threat from nuclear weapons, which we
    have lived with now for more than 50 years, the US Senate's recent
    rejection of the Comprehensive Test Ban Treaty makes it clear
    relinquishing nuclear weapons will not be politically easy. But we
    have a unique opportunity, with the end of the Cold War, to avert a
    multipolar arms race. Building on the BWC and CWC relinquishments,
    successful abolition of nuclear weapons could help us build toward a
    habit of relinquishing dangerous technologies. (Actually, by getting
    rid of all but 100 nuclear weapons worldwide - roughly the total
    destructive power of World War II and a considerably easier task - we
    could eliminate this extinction threat. [30]13)

    Verifying relinquishment will be a difficult problem, but not an
    unsolvable one. We are fortunate to have already done a lot of
    relevant work in the context of the BWC and other treaties. Our major
    task will be to apply this to technologies that are naturally much
    more commercial than military. The substantial need here is for
    transparency, as difficulty of verification is directly proportional
    to the difficulty of distinguishing relinquished from legitimate

    I frankly believe that the situation in 1945 was simpler than the one
    we now face: The nuclear technologies were reasonably separable into
    commercial and military uses, and monitoring was aided by the nature
    of atomic tests and the ease with which radioactivity could be
    measured. Research on military applications could be performed at
    national laboratories such as Los Alamos, with the results kept secret
    as long as possible.

    The GNR technologies do not divide clearly into commercial and
    military uses; given their potential in the market, it's hard to
    imagine pursuing them only in national laboratories. With their
    widespread commercial pursuit, enforcing relinquishment will require a
    verification regime similar to that for biological weapons, but on an
    unprecedented scale. This, inevitably, will raise tensions between our
    individual privacy and desire for proprietary information, and the
    need for verification to protect us all. We will undoubtedly encounter
    strong resistance to this loss of privacy and freedom of action.

    Verifying the relinquishment of certain GNR technologies will have to
    occur in cyberspace as well as at physical facilities. The critical
    issue will be to make the necessary transparency acceptable in a world
    of proprietary information, presumably by providing new forms of
    protection for intellectual property.

    Verifying compliance will also require that scientists and engineers
    adopt a strong code of ethical conduct, resembling the Hippocratic
    oath, and that they have the courage to whistleblow as necessary, even
    at high personal cost. This would answer the call - 50 years after
    Hiroshima - by the Nobel laureate Hans Bethe, one of the most senior
    of the surviving members of the Manhattan Project, that all scientists
    "cease and desist from work creating, developing, improving, and
    manufacturing nuclear weapons and other weapons of potential mass
    destruction."[31]14 In the 21st century, this requires vigilance and
    personal responsibility by those who would work on both NBC and GNR
    technologies to avoid implementing weapons of mass destruction and
    knowledge-enabled mass destruction.

    Thoreau also said that we will be "rich in proportion to the number of
    things which we can afford to let alone." We each seek to be happy,
    but it would seem worthwhile to question whether we need to take such
    a high risk of total destruction to gain yet more knowledge and yet
    more things; common sense says that there is a limit to our material
    needs - and that certain knowledge is too dangerous and is best

    Neither should we pursue near immortality without considering the
    costs, without considering the commensurate increase in the risk of
    extinction. Immortality, while perhaps the original, is certainly not
    the only possible utopian dream.

    I recently had the good fortune to meet the distinguished author and
    scholar Jacques Attali, whose bookLignes d'horizons (Millennium, in
    the English translation) helped inspire the Java and Jini approach to
    the coming age of pervasive computing, as previously described in this
    magazine. In his new bookFraternités, Attali describes how our dreams
    of utopia have changed over time:

    "At the dawn of societies, men saw their passage on Earth as nothing
    more than a labyrinth of pain, at the end of which stood a door
    leading, via their death, to the company of gods and toEternity. With
    the Hebrews and then the Greeks, some men dared free themselves from
    theological demands and dream of an ideal City whereLiberty would
    flourish. Others, noting the evolution of the market society,
    understood that the liberty of some would entail the alienation of
    others, and they soughtEquality."

    Jacques helped me understand how these three different utopian goals
    exist in tension in our society today. He goes on to describe a fourth
    utopia,Fraternity, whose foundation is altruism. Fraternity alone
    associates individual happiness with the happiness of others,
    affording the promise of self-sustainment.

    This crystallized for me my problem with Kurzweil's dream. A
    technological approach to Eternity - near immortality through robotics
    - may not be the most desirable utopia, and its pursuit brings clear
    dangers. Maybe we should rethink our utopian choices.

    Where can we look for a new ethical basis to set our course? I have
    found the ideas in the book Ethics for the New Millennium, by the
    Dalai Lama, to be very helpful. As is perhaps well known but little
    heeded, the Dalai Lama argues that the most important thing is for us
    to conduct our lives with love and compassion for others, and that our
    societies need to develop a stronger notion of universal
    responsibility and of our interdependency; he proposes a standard of
    positive ethical conduct for individuals and societies that seems
    consonant with Attali's Fraternity utopia.

    The Dalai Lama further argues that we must understand what it is that
    makes people happy, and acknowledge the strong evidence that neither
    material progress nor the pursuit of the power of knowledge is the key
    - that there are limits to what science and the scientific pursuit
    alone can do.

    Our Western notion of happiness seems to come from the Greeks, who
    defined it as "the exercise of vital powers along lines of excellence
    in a life affording them scope." [32]15

    Clearly, we need to find meaningful challenges and sufficient scope in
    our lives if we are to be happy in whatever is to come. But I believe
    we must find alternative outlets for our creative forces, beyond the
    culture of perpetual economic growth; this growth has largely been a
    blessing for several hundred years, but it has not brought us
    unalloyed happiness, and we must now choose between the pursuit of
    unrestricted and undirected growth through science and technology and
    the clear accompanying dangers.

    It is now more than a year since my first encounter with Ray Kurzweil
    and John Searle. I see around me cause for hope in the voices for
    caution and relinquishment and in those people I have discovered who
    are as concerned as I am about our current predicament. I feel, too, a
    deepened sense of personal responsibility - not for the work I have
    already done, but for the work that I might yet do, at the confluence
    of the sciences.

    But many other people who know about the dangers still seem strangely
    silent. When pressed, they trot out the "this is nothing new" riposte
    - as if awareness of what could happen is response enough. They tell
    me, There are universities filled with bioethicists who study this
    stuff all day long. They say, All this has been written about before,
    and by experts. They complain, Your worries and your arguments are
    already old hat.

    I don't know where these people hide their fear. As an architect of
    complex systems I enter this arena as a generalist. But should this
    diminish my concerns? I am aware of how much has been written about,
    talked about, and lectured about so authoritatively. But does this
    mean it has reached people? Does this mean we can discount the dangers
    before us?

    Knowing is not a rationale for not acting. Can we doubt that knowledge
    has become a weapon we wield against ourselves?

    The experiences of the atomic scientists clearly show the need to take
    personal responsibility, the danger that things will move too fast,
    and the way in which a process can take on a life of its own. We can,
    as they did, create insurmountable problems in almost no time flat. We
    must do more thinking up front if we are not to be similarly surprised
    and shocked by the consequences of our inventions.

    My continuing professional work is on improving the reliability of
    software. Software is a tool, and as a toolbuilder I must struggle
    with the uses to which the tools I make are put. I have always
    believed that making software more reliable, given its many uses, will
    make the world a safer and better place; if I were to come to believe
    the opposite, then I would be morally obligated to stop this work. I
    can now imagine such a day may come.

    This all leaves me not angry but at least a bit melancholic.
    Henceforth, for me, progress will be somewhat bittersweet.

    Do you remember the beautiful penultimate scene in Manhattan where
    Woody Allen is lying on his couch and talking into a tape recorder? He
    is writing a short story about people who are creating unnecessary,
    neurotic problems for themselves, because it keeps them from dealing
    with more unsolvable, terrifying problems about the universe.

    He leads himself to the question, "Why is life worth living?" and to
    consider what makes it worthwhile for him: Groucho Marx, Willie Mays,
    the second movement of the Jupiter Symphony, Louis Armstrong's
    recording of "Potato Head Blues," Swedish movies, Flaubert's
    Sentimental Education, Marlon Brando, Frank Sinatra, the apples and
    pears by Cézanne, the crabs at Sam Wo's, and, finally, the
    showstopper: his love Tracy's face.

    Each of us has our precious things, and as we care for them we locate
    the essence of our humanity. In the end, it is because of our great
    capacity for caring that I remain optimistic we will confront the
    dangerous issues now before us.

    My immediate hope is to participate in a much larger discussion of the
    issues raised here, with people from many different backgrounds, in
    settings not predisposed to fear or favor technology for its own sake.

    As a start, I have twice raised many of these issues at events
    sponsored by the Aspen Institute and have separately proposed that the
    American Academy of Arts and Sciences take them up as an extension of
    its work with the Pugwash Conferences. (These have been held since
    1957 to discuss arms control, especially of nuclear weapons, and to
    formulate workable policies.)

    It's unfortunate that the Pugwash meetings started only well after the
    nuclear genie was out of the bottle - roughly 15 years too late. We
    are also getting a belated start on seriously addressing the issues
    around 21st-century technologies - the prevention of knowledge-enabled
    mass destruction - and further delay seems unacceptable.

    So I'm still searching; there are many more things to learn. Whether
    we are to succeed or fail, to survive or fall victim to these
    technologies, is not yet decided. I'm up late again - it's almost 6
    am. I'm trying to imagine some better answers, to break the spell and
    free them from the stone.

    1 The passage Kurzweil quotes is from Kaczynski's Unabomber Manifesto,
    which was published jointly, under duress, byThe New York Times and
    The Washington Post to attempt to bring his campaign of terror to an
    end. I agree with David Gelernter, who said about their decision:

    "It was a tough call for the newspapers. To say yes would be giving in
    to terrorism, and for all they knew he was lying anyway. On the other
    hand, to say yes might stop the killing. There was also a chance that
    someone would read the tract and get a hunch about the author; and
    that is exactly what happened. The suspect's brother read it, and it
    rang a bell.

    "I would have told them not to publish. I'm glad they didn't ask me. I

    (Drawing Life: Surviving the Unabomber. Free Press, 1997: 120.)

    2 Garrett, Laurie.The Coming Plague: Newly Emerging Diseases in a
    World Out of Balance. Penguin, 1994: 47-52, 414, 419, 452.

    3 Isaac Asimov described what became the most famous view of ethical
    rules for robot behavior in his bookI, Robot in 1950, in his Three
    Laws of Robotics: 1. A robot may not injure a human being, or, through
    inaction, allow a human being to come to harm. 2. A robot must obey
    the orders given it by human beings, except where such orders would
    conflict with the First Law. 3. A robot must protect its own
    existence, as long as such protection does not conflict with the First
    or Second Law.

    4 Michelangelo wrote a sonnet that begins:

    Non ha l' ottimo artista alcun concetto
    Ch' un marmo solo in sè non circonscriva
    Col suo soverchio; e solo a quello arriva
    La man che ubbidisce all' intelleto.

    Stone translates this as:

    The best of artists hath no thought to show
    which the rough stone in its superfluous shell
    doth not include; to break the marble spell
    is all the hand that serves the brain can do.

    Stone describes the process: "He was not working from his drawings or
    clay models; they had all been put away. He was carving from the
    images in his mind. His eyes and hands knew where every line, curve,
    mass must emerge, and at what depth in the heart of the stone to
    create the low relief."

    (The Agony and the Ecstasy. Doubleday, 1961: 6, 144.)

    5 First Foresight Conference on Nanotechnology in October 1989, a talk
    titled "The Future of Computation." Published in Crandall, B. C. and
    James Lewis, editors.Nanotechnology: Research and Perspectives. MIT
    Press, 1992: 269. See

    6 In his 1963 novelCat's Cradle, Kurt Vonnegut imagined a
    gray-goo-like accident where a form of ice called ice-nine, which
    becomes solid at a much higher temperature, freezes the oceans.

    7 Kauffman, Stuart. "Self-replication: Even Peptides Do It." Nature,
    382, August 8, 1996: 496.

    8 Else, Jon.The Day After Trinity: J. Robert Oppenheimer and The
    Atomic Bomb (available at [35]www.pyramiddirect.com).

    9 This estimate is in Leslie's bookThe End of the World: The Science
    and Ethics of Human Extinction, where he notes that the probability of
    extinction is substantially higher if we accept Brandon Carter's
    Doomsday Argument, which is, briefly, that "we ought to have some
    reluctance to believe that we are very exceptionally early, for
    instance in the earliest 0.001 percent, among all humans who will ever
    have lived. This would be some reason for thinking that humankind will
    not survive for many more centuries, let alone colonize the galaxy.
    Carter's doomsday argument doesn't generate any risk estimates just by
    itself. It is an argument forrevising the estimates which we generate
    when we consider various possible dangers." (Routledge, 1996: 1, 3,

    10 Clarke, Arthur C. "Presidents, Experts, and Asteroids."Science,
    June 5, 1998. Reprinted as "Science and Society" inGreetings,
    Carbon-Based Bipeds! Collected Essays, 1934-1998. St. Martin's Press,
    1999: 526.

    11 And, as David Forrest suggests in his paper "Regulating
    Nanotechnology Development," available
    at[36]www.foresight.org/NanoRev/Forrest1989.html, "If we used strict
    liability as an alternative to regulation it would be impossible for
    any developer to internalize the cost of the risk (destruction of the
    biosphere), so theoretically the activity of developing nanotechnology
    should never be undertaken." Forrest's analysis leaves us with only
    government regulation to protect us - not a comforting thought.

    12 Meselson, Matthew. "The Problem of Biological Weapons."
    Presentation to the 1,818th Stated Meeting of the American Academy of
    Arts and Sciences, January 13, 1999.

    13 Doty, Paul. "The Forgotten Menace: Nuclear Weapons Stockpiles Still
    Represent the Biggest Threat to Civilization."Nature, 402, December 9,
    1999: 583.

    14 See also Hans Bethe's 1997 letter to President Clinton, at

    15 Hamilton, Edith.The Greek Way. W. W. Norton & Co., 1942: 35.

    Bill Joy, cofounder and Chief Scientist of Sun Microsystems, was
    cochair of the presidential commission on the future of IT research,
    and is coauthor ofThe Java Language Specification. His work on
    the[39]Jini pervasive computing technology was featured inWired 6.08.


   39. http://www.wired.com/wired/archive/6.08/jini.html

More information about the paleopsych mailing list