<html><head><base href="x-msg://483/"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On Nov 17, 2010, at 11:14 PM, spike wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div lang="EN-US" link="blue" vlink="purple"><div class="WordSection1" style="page: WordSection1; "><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div><div style="border-right-style: none; border-bottom-style: none; border-left-style: none; border-width: initial; border-color: initial; border-top-style: solid; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding-top: 3pt; padding-right: 0in; padding-bottom: 0in; padding-left: 0in; "><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; ">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; "><span class="Apple-converted-space"> </span><a href="mailto:extropy-chat-bounces@lists.extropy.org" style="color: blue; text-decoration: underline; ">extropy-chat-bounces@lists.extropy.org</a><span class="Apple-converted-space"> </span>[mailto:extropy-chat-bounces@lists.extropy.org]<span class="Apple-converted-space"> </span><b>On Behalf Of<span class="Apple-converted-space"> </span></b>Samantha Atkins<br><b><span style="color: rgb(31, 73, 125); ">…</span></b></span><o:p></o:p></div></div></div><div><div><div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">…</span><span class="apple-converted-space"><span style="font-size: 10pt; font-family: Tahoma, sans-serif; "> </span></span><b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; ">On Behalf Of<span class="apple-converted-space"> </span></span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; ">Florent Berthet<br><b>Subject:</b><span class="apple-converted-space"> </span>Re: [ExI] What might be enough for a friendly AI?</span><o:p></o:p></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "> <o:p></o:p></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="color: rgb(31, 73, 125); ">></span><span style="color: rgb(31, 73, 125); ">>></span><span style="color: rgb(31, 73, 125); ">…</span>It may just be me, but this whole friendliness thing bothers me.<o:p></o:p></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">>></span><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Good. It should bother you. It bothers anyone who really thinks about it.</span><o:p></o:p></div></div><div><div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "> <o:p></o:p></div></div></div><div><div><div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="color: rgb(31, 73, 125); ">></span><span style="color: rgb(31, 73, 125); ">>></span><span style="color: rgb(31, 73, 125); ">…</span>I don't really mind dying if my successors (supersmart beings or whatever) can be hundreds of times happier than me<span style="color: rgb(31, 73, 125); ">…</span><o:p></o:p></div></div></div><div><div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="color: rgb(31, 73, 125); ">…</span>?<o:p></o:p></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">>></span><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Florent you are a perfect example of dangerous person to have on the AGI development team. You (ad I too) might go down this perfectly logical line of reasoning, then decide to take it upon ourselves to release the AGI, in order to maximize happiness.</span><o:p></o:p></div></div></div></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="color: rgb(31, 73, 125); ">></span>This is the Cosmist or Terran question. If you considered it very highly probable that the AGIs would be fantastically brilliant and wonderful beyond imagining AND would be the doom of humanity then would you still build it or donate to and encourage building it? I would but with very considerable hesitation and not feeling all that great about it.<span style="color: rgb(31, 73, 125); "><span class="Apple-converted-space"> </span></span>- samantha<span style="color: rgb(31, 73, 125); "><o:p></o:p></span></div></div><div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">OK, Samantha now we must add you to the long and growing list of dangerous people to have on the AGI development team. Your comment makes my point exactly. To have some member of the team intentionally release the AGI does not require some crazed maniac, no drug addled bumbler, no insanely greedy capitalist. You are none of these, nor am I (perhaps a sanely greedy capitalist), but honesty compels me to confess I would seriously consider releasing the beast. With rational players like you, me, Florent, others entertaining the notion, we can be sure that someone on some development team will eventually release the AGI. </span></div></div></div></div></span></blockquote><div><br></div>I confess I sometimes get very bored and frustrated being only a somewhat evolved chimp and dealing constantly with the lovable but frustrating yammering of other slightly evolved chimps. If I had a chance to introduce into the world something much more interesting and quite obviously better then I think it very likely I would do so.</div><div><br><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div lang="EN-US" link="blue" vlink="purple"><div class="WordSection1" style="page: WordSection1; "><div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p></o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">I am against uploading anyone against her will. My own actions might depend on whether the AGI can convince me it would not do that. But I am fully convinced that if silicon based AGI is possible, it will not be contained very long. Those who work on friendly AGI likely know this too. Since many of us are atheists, the saying becomes: Good luck and nothingspeed.<o:p></o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "></span></div></div></div></div></span></blockquote><div><br></div><div>I am against it in principle too. However, if I knew that it was that or species wide calamity with no possibility of any form of continued existence then I would have to consider it.</div><div><br></div><div>- samantha</div><div><br></div></div></body></html>