[extropy-chat] Eugen Leitl on AI design

[Jef Allbright]

Eliezer Yudkowsky wrote:

> Jeff Davis wrote:
>
>>
>> I am of the "intelligence leads inevitably to ethics"
>> school.  (I consider ethics a form of advanced
>> rationality.  Which springs from the modeling and
>> symbol manipulation emblematic of the quality which we
>> fuzzily refer to as intelligence.)  It has done so
>> with humans, where the "intelligence"--such as it is,
>> puny not "super"--has evolved from the mechanical
>> randomness and cold indifference of material reality.
>
>
> I too considered morality a special case of rationality, back in 
> 1996-2000 before I understood exactly how it all worked.  It's an easy 
> enough mistake to make.  But the math says rationality is a special 
> case of morality, not the other way around; and rationality can be a 
> special case of other moralities than ours.

<snip>

>
> Everyone please recall that I started out confidently stating "The 
> Powers will be ethical!" and then moved from that position to this 
> one, driven by overwhelmingly strong arguments.  It shouldn't have 
> taken overwhelmingly strong arguments, and next time I shall endeavor 
> to allow my beliefs to be blown about like leaves on the winds of 
> evidence, and also not make confident statements about anything before 
> I understand the fundamental processes at work.  But the 
> overwhelmingly strong reasons that drove me to this position are 
> there, even if most of them are hard to explain.

Intelligence is one of the pillars of morality.  Another pillar is 
interdependence.  Another, even more subtle, is growth.

Wasn't so long ago, in the evolution of humanist thought, that 
*intelligence* was first seen as the beacon of enlightenment that would 
allow humanity to move beyond the previous confines of religion and 
superstition.  "Free thinkers" made great progress and patted themselves 
on the back for how smart they were.  Naturally they applied this 
powerful concept to everything they could, and impressed with the 
revolutionary progress they had made, extrapolated that all of 
humanity's questions could be best answered via the application of 
rational intelligence.  I think they were right, within the context of 
their awareness.

A few of these rational free thinkers sensed that there was still 
something missing.  Rationality is bounded by knowledge, and a new level 
of enlightenment arose in which people began to realize a need for 
wisdom within uncertainty.  Some of these people were mistaken for 
mystics, but rather than abandoning rational thought, these newer 
thinkers worked to incorporate rational thinking into a larger framework 
that acknowledged, and even welcomed uncertainty.  Mathematical 
statistics (of the frequentist sort and more recently Bayesian) were 
joined by newer concepts of entropy and theories of information and 
incompleteness, and there was a pervasive belief among rational 
free-thinkers that if humanity just learned the right equations, they 
could understand the universe.  And great strides were made in many 
technological areas, and they were right, within the context of their 
awareness.

More recently, concepts of uncertainty and randomness are being 
overtaken by ideas of chaos and complexity, and rational free-thinkers 
are discovering some of the inherent  limits of modeling and prediction 
with finite computational resources.  We're finding that much of the 
really interesting stuff can't be modeled or predicted and the only way 
to determine the end result is to actually play it out.  *This changes 
the focus of the game away from modeling and extrapolation, and towards 
understanding  what freedoms (points of influence) are available to us 
in order to create an always evolving and unpredictable future.*  These 
new concepts do not replace, but encompass and extend the previous paradigm.

I offer this as a necessarily abbreviated and simplified history of the 
development of rational thinking on the human scale, and also perhaps 
the development of individual thinking among members of this list 
growing up within that knowledge environment.  Although overstated, 
perhaps "ontogeny recapitulates phylogeny" applies here as well.

- Jef

http://www.jefallbright.net