[extropy-chat] Fools building AIs (was: Tyranny in place)

[Eugen Leitl]

On Fri, Oct 06, 2006 at 11:53:30AM -0400, Ben Goertzel wrote:

> No... it just doesn't GUARANTEE a value being placed on self-preservation...

I still don't know what you mean when you use 'rational'.
http://en.wikipedia.org/wiki/Rationality says several things, some of them
being
"In economics, sociology, and political science, a decision or situation is often called rational if it is in some sense optimal, and individuals or organizations are often called rational if they tend to act somehow optimally in pursuit of their goals. Thus one speaks, for example, of a rational allocation of resources, or of a rational corporate strategy. In this concept of "rationality", the individual's goals or motives are taken for granted and not made subject to criticism, ethical or otherwise. Thus rationality simply refers to the success of goal attainment, whatever those goals may be. Sometimes, in this context, rationality is equated with behavior that is self-interested to the point of being selfish. Sometimes rationality implies having complete knowledge about all the details of a given situation. It might be said that because the goals are not important in definition of rationality, it really only demands logical consistency in choice making. See rational choice theory.

Debates arise in these three fields about whether or not people or organizations are "really" rational, as well as whether it make sense to model them as such in formal models. Some have argued that a kind of bounded rationality makes more sense for such models. Others think that any kind of rationality along the lines of rational choice theory is a useless concept for understanding human behavior; the term homo economicus (economic man: the imaginary logically consistent but amoral being assumed in economic models) was coined largely in honor of this view.

Sociologist Max Weber's writing can be interpreted as suggesting an increasing irrationality of rationality.

Rationality is a central principle in artificial intelligence, where a rational agent is specifically defined as an agent which always chooses the action which maximises its expected performance, given all of the knowledge it currently possesses."

Perhaps you want to define your use of the term.

-- 
Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org
______________________________________________________________
ICBM: 48.07100, 11.36820            http://www.ativel.com
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 191 bytes
Desc: Digital signature
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20061006/9a526325/attachment.bin>