<HTML><BODY style="word-wrap: break-word; -khtml-nbsp-mode: space; -khtml-line-break: after-white-space; "><BR><DIV><DIV>On May 16, 2006, at 1:39 PM, Russell Wallace wrote:</DIV><BR class="Apple-interchange-newline"><BLOCKQUOTE type="cite">On 5/16/06, <B class="gmail_sendername">Eliezer S. Yudkowsky</B> <<A href="mailto:sentience@pobox.com">sentience@pobox.com</A>> wrote:<DIV><SPAN class="gmail_quote"></SPAN><BLOCKQUOTE class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> It is rather reminiscent of someone lecturing me on how, if I don't<BR>believe in Christ, Christ will damn me to hell. But Christians have at<BR>least the excuse of being around numerous other people who all believe<BR>exactly the same thing, so that they are no longer capable of noticing <BR>the dependency on their assumptions, or of properly comprehending that<BR>another might share their assumptions.</BLOCKQUOTE><DIV><BR> *ahem* Given that in both cases I'm the one _failing_ to believe in the imminent manifestation of deity on Earth via never-observed processes, I think your analogy has one of those "minus sign switched with plus sign" bugs ;)<BR></DIV></DIV></BLOCKQUOTE><DIV><BR class="khtml-block-placeholder"></DIV>You seem to be confusing Singularity with some particular extremely utopian or dystopian outcomes. I am not using the word in that sense and I don't think many others in this thread are either. I don't expect a deity to manifest on Earth as part and parcel of Singularity.</DIV><DIV><BR><BLOCKQUOTE type="cite"><DIV><DIV> </DIV><BR><BLOCKQUOTE class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">And you, Russell, need not list any of the negative consequences of<BR>believing in a hard takeoff when it doesn't happen, to convince me that <BR>it would be well not to believe in it supposing it doesn't happen.<BR> </BLOCKQUOTE></DIV><BR> Then perhaps what I need to do is explain why it won't happen? Okay, to make sure we're on the same page, a definition:<BR> <BR> Hard takeoff = a process by which seed AI (of complexity buildable by a team of very smart humans in a basement) undertakes self-improvement to increase its intelligence, without needing mole quantities of computronium, _without needing to interact with the real world outside its basement_, and without needing a long time; subsequently emerging in a form that is already superintelligent.<BR> <BR></BLOCKQUOTE><DIV><BR class="khtml-block-placeholder"></DIV><DIV>Who says this will happen with no interaction with the rest of the world on the part of the seed AI? As you know much has been written about the difficulty of keeping the AI isolated sufficiently from the world. But I have no reason to consider self-improvement or even really clever hacking utterly insufficient to reach super-intelligence in that "basement" with just the corpus of information reachable in read-only mode from the internet and reasonably good self-improving code. The latter is imho key if in fact humans are incapable of designing the components of a super-intelligence. Why do you think computronium is required? It was not required to get to human intelligence obviously.</DIV><BR><BLOCKQUOTE type="cite"> 1) I've underlined the most important condition, because it exposes the critical flaw: the concept of hard takeoff assumes "intelligence" is a formal property of a collection of bits. It isn't. It's an informal comment on the relationship between a collection of bits and its environment.<BR></BLOCKQUOTE><DIV><BR class="khtml-block-placeholder"></DIV>Huh? It is you who posited a completely isolated environment that cannot be breached. An initial environment does not have to include the entire world in order for the intelligence to grow. </DIV><DIV><BR><BLOCKQUOTE type="cite"> <BR> Suppose your AI comes up with a new, supposedly more intelligent version of itself and is trying to decide whether to replace the current version with the new version. What algorithm does it use to tell it whether the new version is really more intelligent? Well it could apply some sort of mathematical criterion, see whether the new version is faster at proving a list of theorems, say.<BR> <BR></BLOCKQUOTE><DIV><BR class="khtml-block-placeholder"></DIV><DIV>Or however long a list of skills you consider markers of intelligence that you care to name. But all of that would not be the point. If the AI can improve the very basis of intelligence, the very components of its mind, then it will become more intelligent over time. Your argument is not convincing that this is impossible. </DIV><BR><BLOCKQUOTE type="cite"> But the most you will ever get out of that process is an AI that's intelligent at proving mathematical theorems - not because you didn't apply it well enough, but because that's all the process was ever trying to give you in the first place. To create an AI that's intelligent at solving real world problems - one that can cure cancer or design a fusion reactor that'll actually work when it's turned on - requires that the criterion for checking whether a new version is really more intelligent than the old one, involves testing its effectiveness at solving real world problems. Which means the process of AI development must involve interaction with the real world, and must be limited in speed by real world events even if you have a building full of nanocomputers to run the software.<BR> <BR></BLOCKQUOTE><DIV><BR class="khtml-block-placeholder"></DIV><DIV>Yes and no. What is the real world constraint on building and testing software? Yes if you want to solve engineering problems you eventually need to get beyond the design tools and simulation but these are still highly critical. Actually building and testing a new solution can be done by human beings from the design the AI came up with. Where is the crucial problem?</DIV><DIV><BR class="khtml-block-placeholder"></DIV><BR><BLOCKQUOTE type="cite"> There are other problems, mind you:<BR> <BR> 2) Recursive self-improvement mightn't be a valid concept in the first place. If you think about it, our reason for confidence in the possibility of AI and nanotechnology is that we have - we are - existence proofs. There is no existence proof for full RSI. On the contrary, all the data goes the other way: every successful complex system we see, from molecular biology to psychology, law and politics, computers and the Internet etc, are designed as a stack of layers where the upper layers are easy to change, but they rest on more fundamental lower ones that typically are changed only by action of some still more fundamental environment.<BR> <BR></BLOCKQUOTE><DIV><BR class="khtml-block-placeholder"></DIV><DIV>I don't see this as valid. Code can be optimized and re-factored by code. Code can write, profile, redesign and rewrite code. If humans can improve code with our obvious lack of specialization for such tasks I have no reason to believe that it is impossible to capture the relevant knowledge and design a software system to do so. </DIV><BR><BLOCKQUOTE type="cite"><BR> The matter has been put to the test: Eurisko was a full RSI <BR> AI. Result: it needed constant human guidance or it'd just wedge itself.<BR> <BR></BLOCKQUOTE><DIV><BR class="khtml-block-placeholder"></DIV><DIV>So one early effort has exhausted the entire problem space?</DIV><BR><BLOCKQUOTE type="cite"> 3) On to the most trivial matter, computing power: there's been a lot of discussion about how much it might take to run a fully developed and optimized AI, and the consensus is that ultimate silicon technology might suffice... but that line of argument ignores the fact that it takes much more computing power to develop an algorithm than it does to run it once developed.<BR></BLOCKQUOTE><DIV><BR class="khtml-block-placeholder"></DIV><DIV>How is this relevant? Are you including all discovery of the problem space, design efforts, checking the results and so on? So what? Does this mean it is impossible? Not at all.</DIV><BLOCKQUOTE type="cite"><BR> Bottom line, I think we'll need molecular computers to develop AI - and not small prototype ones either.<BR> <BR></BLOCKQUOTE><DIV><BR class="khtml-block-placeholder"></DIV><DIV>I don't.</DIV><BR><BLOCKQUOTE type="cite"> So that's one reason why the concept is definitely incoherent, an independent reason why there's good reason to believe it's incoherent, and a third again independent reason why it wouldn't be practical in the foreseeable future even if it were coherent.<BR></BLOCKQUOTE><DIV><BR class="khtml-block-placeholder"></DIV><DIV><BR class="khtml-block-placeholder"></DIV>That you think a highly contentious thing is true (but you cannot prove it) cannot be used to say "definitely" that something else is the case. Your entire post amounts to you merely saying it can't be done without proving any such thing or imho even making a good case.</DIV><DIV><BR class="khtml-block-placeholder"></DIV><DIV>- samantha</DIV><BR class="khtml-block-placeholder"></BODY></HTML>