<HTML><FONT FACE=arial,helvetica><HTML><FONT SIZE=2 PTSIZE=10>In a message dated 06/03/2008 21:55:36 GMT Standard Time, rpwl@lightlink.com writes:<BR>
<BR>
<BR>
<BLOCKQUOTE TYPE=CITE style="BORDER-LEFT: #0000ff 2px solid; MARGIN-LEFT: 5px; MARGIN-RIGHT: 0px; PADDING-LEFT: 5px">This line of argument makes the following assumption:<BR>
<BR>
*** Any AGI sufficiently intelligent to be a threat would start off <BR>
in such a state that its drive system (its motivations or goals, to <BR>
speak loosely) would either be unknowable by us, or deliberately <BR>
programmed to be malicious, or so unstable that they would quickly <BR>
deviate from heir initial set.<BR>
</FONT><FONT COLOR="#000000" BACK="#ffffff" style="BACKGROUND-COLOR: #ffffff" SIZE=2 PTSIZE=10 FAMILY="SANSSERIF" FACE="Arial" LANG="0"></BLOCKQUOTE><BR>
<BR>
</FONT><FONT COLOR="#000000" BACK="#ffffff" style="BACKGROUND-COLOR: #ffffff" SIZE=2 PTSIZE=10 FAMILY="SANSSERIF" FACE="Arial" LANG="0">Unknowable by us, most probable. We would have to control each and every piece of data it receives and calculate every reaction to that data in order to know with certainty its motivations (motivation in these terms, logically determined actions in my terms). A mathematically insurmountable task.<BR>
Deliberately malicious. I don't agree that this would need to be so. If anything this is a concern that <U>is</U> a possibility. Hackers are generally new adopters of all technology and such juvenile tinkering could well result in deliberately malicious programming or simply through pure ignorance, derailing a friendly AI. I don't need to highlight the possible military implications regarding desirability of a malicious AI.<BR>
Unstable may not be the clearest term I would use. Certainly unstable from our point of view, but more probably the learning curve of the AI would be so stochastic that we cannot calculate the outcome. This would be true of its learned logic, knowledge base and any psydo-emotions which it may have. The end result is a chaotic erratic system (from our eyes) which would be impossible to predict.<BR>
</FONT><FONT COLOR="#000000" BACK="#ffffff" style="BACKGROUND-COLOR: #ffffff" SIZE=2 PTSIZE=10 FAMILY="SANSSERIF" FACE="Arial" LANG="0"><BR>
</FONT><FONT COLOR="#000000" BACK="#ffffff" style="BACKGROUND-COLOR: #ffffff" SIZE=2 PTSIZE=10 FAMILY="SANSSERIF" FACE="Arial" LANG="0"><BLOCKQUOTE TYPE=CITE style="BORDER-LEFT: #0000ff 2px solid; MARGIN-LEFT: 5px; MARGIN-RIGHT: 0px; PADDING-LEFT: 5px">This assumption is massively dependant on the actual design of the AGI <BR>
itself. Nobody can state that an AGI would behave in this or that way <BR>
without being very specific about the design of the AGI they are talking <BR>
about.<BR>
</FONT><FONT COLOR="#000000" BACK="#ffffff" style="BACKGROUND-COLOR: #ffffff" SIZE=2 PTSIZE=10 FAMILY="SANSSERIF" FACE="Arial" LANG="0"></BLOCKQUOTE><BR>
<BR>
</FONT><FONT COLOR="#000000" BACK="#ffffff" style="BACKGROUND-COLOR: #ffffff" SIZE=2 PTSIZE=10 FAMILY="SANSSERIF" FACE="Arial" LANG="0">Agreed, as per above.<BR>
</FONT><FONT COLOR="#000000" BACK="#ffffff" style="BACKGROUND-COLOR: #ffffff" SIZE=2 PTSIZE=10 FAMILY="SANSSERIF" FACE="Arial" LANG="0"><BR>
</FONT><FONT COLOR="#000000" BACK="#ffffff" style="BACKGROUND-COLOR: #ffffff" SIZE=2 PTSIZE=10 FAMILY="SANSSERIF" FACE="Arial" LANG="0"><BLOCKQUOTE TYPE=CITE style="BORDER-LEFT: #0000ff 2px solid; MARGIN-LEFT: 5px; MARGIN-RIGHT: 0px; PADDING-LEFT: 5px">The problem is that many people assume a design for the AGI's motivation <BR>
system that is theoretically untenable. To be blunt, it just won't <BR>
work. There are a variety of reasons why it won't work, but regardless <BR>
of what those reasons actually are, the subject of any discussion of <BR>
what an AGI "would" do has to be a discussion of its motivation-system <BR>
design.<BR>
<BR>
By contrast, most discussions I have seen are driven by wild, <BR>
unsupported assertions about what an AGI would do! Either that, or they <BR>
contain assertions about ideas that are supposed to be real threats (see <BR>
the list above) which are actually trivially easy to avoid or deeply <BR>
unlikely.<BR>
</FONT><FONT COLOR="#000000" BACK="#ffffff" style="BACKGROUND-COLOR: #ffffff" SIZE=2 PTSIZE=10 FAMILY="SANSSERIF" FACE="Arial" LANG="0"></BLOCKQUOTE><BR>
<BR>
</FONT><FONT COLOR="#000000" BACK="#ffffff" style="BACKGROUND-COLOR: #ffffff" SIZE=2 PTSIZE=10 FAMILY="SANSSERIF" FACE="Arial" LANG="0">Pointing back to my earlier post I stated:<BR>
<I>An Intelligence of this magnitude with a global reach into just about every control system on the planet could and probably will do major damage. Although probably not through design or desire, but just through exploration of ability or pure accident.<BR>
</I>Even if the AGI were boxed in or only had limited external contact, I can't imagine how we could keep it cooped up for very long.<BR>
<BR>
I can't see how you can reduce the list of threats to '<I>Trivial.'</I> How do you propose we 'easily' avoid them? <I><BR>
<BR>
<BR>
Alex<BR>
</I><BR>
<BR>
</FONT> </HTML>