<HTML><BODY style="word-wrap: break-word; -khtml-nbsp-mode: space; -khtml-line-break: after-white-space; "><BR><DIV><DIV>On Jun 17, 2007, at 1:02 AM, <A href="mailto:extropy-chat-request@lists.extropy.org">extropy-chat-request@lists.extropy.org</A> wrote:</DIV><BR class="Apple-interchange-newline"><BLOCKQUOTE type="cite"><BLOCKQUOTE type="cite"><P style="margin: 0.0px 0.0px 0.0px 10.0px"><FONT face="Helvetica" size="3" style="font: 12.0px Helvetica">It is of course very important that no-one be forced to do anything they don't want to do.</FONT></P> <BR></BLOCKQUOTE><P style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><BR></P> <P style="margin: 0.0px 0.0px 0.0px 0.0px"><FONT face="Helvetica" size="3" style="font: 12.0px Helvetica">Cheap slogan.<SPAN class="Apple-converted-space"> </SPAN>What about five-year-olds?<SPAN class="Apple-converted-space"> </SPAN>Where do you draw the line?</FONT></P> <P style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><BR></P> <P style="margin: 0.0px 0.0px 0.0px 0.0px"><FONT face="Helvetica" size="3" style="font: 12.0px Helvetica">Someone says they want to hotwire their brain's pleasure center; they say they think it'll be fun.<SPAN class="Apple-converted-space"> </SPAN>A nearby AI reads off their brain state and announces unambiguously that they have no idea what'll actually happen to them - they're definitely working based on mistaken expectations.<SPAN class="Apple-converted-space"> </SPAN>They're too stubborn to listen to warnings, and they're picking up the handy neural soldering iron (they're on sale at Wal-Mart, a very popular item).<SPAN class="Apple-converted-space"> </SPAN>What's the moral course of action? For you?<SPAN class="Apple-converted-space"> </SPAN>For society?<SPAN class="Apple-converted-space"> </SPAN>For a superintelligent AI?</FONT></P> <P style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><BR></P> <P style="margin: 0.0px 0.0px 0.0px 0.0px"><FONT face="Helvetica" size="3" style="font: 12.0px Helvetica">--<SPAN class="Apple-converted-space"> </SPAN></FONT></P> <P style="margin: 0.0px 0.0px 0.0px 0.0px"><FONT face="Helvetica" size="3" style="font: 12.0px Helvetica">Eliezer S. Yudkowsky<SPAN class="Apple-converted-space"> </SPAN><A href="http://singinst.org">http://singinst.org</A>/</FONT></P></BLOCKQUOTE><BR></DIV><DIV>Eliezer,</DIV><DIV><BR class="khtml-block-placeholder"></DIV><DIV>Where coercion begins morality ends. The principle of respect for self ownership cannot not fully extend to those who cannot sustain their own existence or to those who choose to disown themselves. I'm speaking in the social context where "self" means an individual that can be distinguished from other individuals rather than the psychological context wherein many here believe "self" is an illusion created by a summation of competing mind agents. Adults have the moral option of protecting the children they value. Adults have the moral right to self destroy. An AI, depending on its programming, might or might not have the option of salvaging a disowned self. If one cannot or will not own oneself, I think salvage rights apply and victim status becomes questionable. Don't you? -- Thomas</DIV><BR><DIV> <DIV style="text-align: center;"><A href="mailto:Thomas@ThomasOliver.net">Thomas@ThomasOliver.net</A></DIV> </DIV><BR></BODY></HTML>