[extropy-chat] AI design

[Eliezer Yudkowsky]

Alejandro Dubrovsky wrote:

> On Thu, 2004-06-03 at 05:57 -0400, Eliezer Yudkowsky wrote:
> 
>>An AI that had been programmed with a static utility function over human 
>>happiness and to not destroy human bodies, would rewrite all extant human 
>>brains in a state of maximum "pleasure" as defined in the utility function, 
>>and then freeze them in that exact position (because the utility function 
>>is over static states rather than dynamic states).
> 
> i don't see why the utility function can't be time dependent. eg
> V(x sub t) = pleasure of system x at time t
> U(x sub t) = { 0 if V(x sub t) in  set (V(x sub t1 where 0 < t1 < t mod
> (number of possible states of x)));
> V(x sub t) otherwise
> }
> 
> Not that i would recommend that as a utility function.

Sure, you can have time-dependent utility functions, though that's not 
exactly what you need; you need a utility function that operates on 
descriptions of the universe's 4D state.  But that also is not sufficient. 
  You can win, but you have to *think of* the problems *in advance* and 
develop a *general* theory to deal with them, not just rely on thinking of 
every single problem in advance.  Hence the PhD in a field that doesn't 
exist.  Meanwhile, as far as I know, I'm the only person thinking about 
this class of problems *at all*.  Remember that thinking about this problem 
class requires realizing that superintelligence is an important domain.  An 
AI that doesn't have the power to tile the universe with tiny smiley faces, 
that can only produce smiley faces by making humans smile, will appear to 
work fine during its childhood.  And thinking also requires realizing that 
the problem is difficult and urgent enough to require thinking through in 
advance, and so on.  And it requires the ability think about possible 
disasters without someone needing to nudge you into it, without going 
mystical, without looking for handy rationalizations why it *might* not 
happen, and so on.  My experience suggests that this is a bar that is not 
only above most AI researchers, it is a bar astronomically above most AI 
researchers.  They never even get close to addressing the difficult and 
interesting challenge; they fail on the earliest possible part of the problem.

So it is not that Earth is inevitably doomed.  The problems are 
individually solvable, and I even think that the meta-problem is solvable, 
so that missing one thing won't kill us.  But Earth is doomed by default, 
and very few people care, or are interested in rising to the challenge.

Incidentally, Dubrovsky, since you replied with math, which is a very rare 
thing, can I ask you to start listing other things that you think might go 
wrong with a Really Powerful Optimization Process?  To see if any of them 
are problems I missed?

-- 
Eliezer S. Yudkowsky                          http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence