<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><base href="x-msg://483/"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.apple-style-span
{mso-style-name:apple-style-span;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> extropy-chat-bounces@lists.extropy.org [mailto:extropy-chat-bounces@lists.extropy.org] <b>On Behalf Of </b>Samantha Atkins<br><b><span style='color:#1F497D'>…</span></b></span><o:p></o:p></p></div></div><div><div><div><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>…</span><span class=apple-converted-space><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> </span></span><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>On Behalf Of<span class=apple-converted-space> </span></span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>Florent Berthet<br><b>Subject:</b><span class=apple-converted-space> </span>Re: [ExI] What might be enough for a friendly AI?</span><o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal><span style='color:#1F497D'>></span><span style='color:#1F497D'>>></span><span style='color:#1F497D'>…</span>It may just be me, but this whole friendliness thing bothers me.<o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>>></span><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Good. It should bother you. It bothers anyone who really thinks about it.</span><o:p></o:p></p></div><div><div><p class=MsoNormal> <o:p></o:p></p></div></div><div><div><div><p class=MsoNormal><span style='color:#1F497D'>></span><span style='color:#1F497D'>>></span><span style='color:#1F497D'>…</span>I don't really mind dying if my successors (supersmart beings or whatever) can be hundreds of times happier than me<span style='color:#1F497D'>…</span><o:p></o:p></p></div></div><div><div><p class=MsoNormal><span style='color:#1F497D'>…</span>?<o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>>></span><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Florent you are a perfect example of dangerous person to have on the AGI development team. You (ad I too) might go down this perfectly logical line of reasoning, then decide to take it upon ourselves to release the AGI, in order to maximize happiness.</span><o:p></o:p></p></div></div></div></div><div><p class=MsoNormal><o:p> </o:p></p></div><p class=MsoNormal><span style='color:#1F497D'>></span>This is the Cosmist or Terran question. If you considered it very highly probable that the AGIs would be fantastically brilliant and wonderful beyond imagining AND would be the doom of humanity then would you still build it or donate to and encourage building it? I would but with very considerable hesitation and not feeling all that great about it.<span style='color:#1F497D'> </span>- samantha<span style='color:#1F497D'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>OK, Samantha now we must add you to the long and growing list of dangerous people to have on the AGI development team. Your comment makes my point exactly. To have some member of the team intentionally release the AGI does not require some crazed maniac, no drug addled bumbler, no insanely greedy capitalist. You are none of these, nor am I (perhaps a sanely greedy capitalist), but honesty compels me to confess I would seriously consider releasing the beast. With rational players like you, me, Florent, others entertaining the notion, we can be sure that someone on some development team will eventually release the AGI. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I am against uploading anyone against her will. My own actions might depend on whether the AGI can convince me it would not do that. But I am fully convinced that if silicon based AGI is possible, it will not be contained very long. Those who work on friendly AGI likely know this too. Since many of us are atheists, the saying becomes: Good luck and nothingspeed.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>spike<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> <o:p></o:p></span></p></div></div></body></html>