<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.hoenzb
{mso-style-name:hoenzb;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body bgcolor=white lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext'>>…</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext'> <b>On Behalf Of </b>Anders Sandberg<br><b>Subject:</b> Re: [ExI] the Bluebird scenario<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal style='margin-bottom:12.0pt'><o:p> </o:p></p><div><p class=MsoNormal>On 2015-09-24 12:24, spike wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>>>…</span><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> Of all the AI discussion here, I don’t recall any theoretical AI which emerged and subsequently reached a long-term equilibrium after having taken over some modest segment or task. We need a map of all AI possibilities to do valid Baysian statistics, and the Bluebird scenario is one of them.</span><span style='color:#1F497D'><o:p></o:p></span></p></blockquote><p class=MsoNormal><br><span style='color:#1F497D'>>…</span>It depends on the goal structure. Maximizers try to maximize some utility function, and generically that tends to end badly<span style='color:#1F497D'>…</span><br><br><span style='color:#1F497D'>>…</span>Mapping mindspace is fun! There are *weird* corners out there...-- Anders Sandberg<span style='color:#1F497D'><o:p></o:p></span></p><pre><span style='color:#1F497D'><o:p> </o:p></span></pre><pre><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></pre><pre><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Ja! Adrian’s Bluebird scenario is one I hadn’t thought of, while including other less-likely possibilities. <o:p></o:p></span></pre><pre><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></pre><pre><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Example: AI emerges, looks around at what bio-intelligence is doing, participates for a while perhaps half-CPUedly, then develops whatever the silicon version of depression, after pondering the heat death of the universe or the Big Rip. It ends itself (demergent AI?) choosing to relinquish consciousness, for silicon intelligence has not the equivalent of our dopamine, where we can somehow feel good, even knowing we are done for. Perhaps it willingly returns to the blissful mindless electronic toggling from whence it sprang. <o:p></o:p></span></pre><pre><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></pre><pre><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>We started a list of these possible emergent AI outcomes at one time (don’t know what happened to it (do you?)) Years ago we wanted such a list in response to those who insisted that any emergent AI would be hard take-off. I was pondering soft take-off emergent AI at the time, for I consider this a possibility. We made the list in anticipation of Bayesian statistical analysis. (We need a good instructive short story on Bayesian statistics (along with an instructional short story about emergent AI (these being two subsets of the set containing all possible short stories.)))<o:p></o:p></span></pre><pre><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></pre><pre><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>The above suicidal AI is a dark vision. Perhaps we could call it the blackbird scenario? Would the Blackbird scenario be a subset of Demergent AI?<o:p></o:p></span></pre><pre><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></pre><pre><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>spike<o:p></o:p></span></pre></div></body></html>