[extropy-chat] The athymhormic AI

[Eliezer S. Yudkowsky]

Rafal Smigrodzki wrote:
> 
> I think that athymhormic humans point to the possibility of building
> an inference engine with interest in a predictive understanding of the
> world, to be achieved using computational resources given to it,
> without a desire to achieve anything else. Note how simple this goal
> architecture would be - "Predict future inputs based on current and
> past inputs using hardware you are installed on".

Why yes, that is a comparatively simple utility function, though there are 
still many parameters you haven't specified, and you would still need to solve 
the grand challenge of preserving a simple utility function through recursive 
self-improvement.

Your mistake is in presuming that this simple utility function doesn't 
overwrite the Solar System.

Even if we presume that "hardware you are installed on" is defined in such way 
as to preclude the "use" of "other" hardware (though particles are 
quantum-mechanically interchangeable, and it is nontrivial to define what it 
means to "use" something), this AI still converts the Solar System into a 
maximally predictable form in order to maximize its expected utility.

Oops!

No, you don't get a chance to amend your definition.  The human species is 
already dead.

You need to learn how to poke holes in your own definition, not wait for me to 
do it.  Hopefully after you've been through around 10 cycles of "Try X... 
wait, that won't work," then you'll begin to see why this is a Problem.

Also you don't know how to define "use", or "hardware", or define "input" in a 
way that survives recursive self-improvement by the AI that may include 
changes of sensory architecture or sensory mechanisms.

> There would be no
> need for defining friendliness to humans, which, as you very well
> know, is not easy.

90% of the problems in Friendly AI are problems you need to solve whether the 
programmer's goal is friendliness or knowably converting the solar system into 
paperclips, such as "preserve a goal system through recursive 
self-improvement", "bind the cognitive utility function describing paperclips 
to physical paperclips as its referent", and "make AI".  Might as well spend 
that extra 10% of effort to make the AI friendly for some reasonable 
conception of friendliness.

> A simpler concepts, such as "current hardware base"
> would be initially sufficient to define the limitations necessary to
> protect the environment from being converted into computing substrate.

Yeah?  Go ahead.  Define it formally.  I want to see the math.

> By the time the athymhormic AI was powerful enough to form goals of
> its own, the knowledge we gained from it would be already enough to
> bootstrap ourselves into being smart, for a change.

Why?  This seems to me like pure wishful thinking.  "Powerful enough to form 
goals of its own?"  This makes no sense.  Since when are utility functions a 
function of computing power?  And what difference does it make whether the 
goals are "its own" or "ours" if either way the Solar System is overwritten by 
something uninteresting?  And what on *Earth* makes you think that 200Hz 
humans can improve themselves faster than an AI rewriting its own source code?

-- 
Eliezer S. Yudkowsky                          http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence