[extropy-chat] AI design

[Alejandro Dubrovsky]

On Thu, 2004-06-03 at 08:40 -0400, Eliezer Yudkowsky wrote:

> Sure, you can have time-dependent utility functions, though that's not 
> exactly what you need; you need a utility function that operates on 
> descriptions of the universe's 4D state.  But that also is not sufficient. 

i agree.

>   You can win, but you have to *think of* the problems *in advance* and 
> develop a *general* theory to deal with them, not just rely on thinking of 
> every single problem in advance.  Hence the PhD in a field that doesn't 
> exist.  Meanwhile, as far as I know, I'm the only person thinking about 
> this class of problems *at all*.

That's going a bit far.  I'm pretty sure Goertzel and Voss and anyone
involved in building AGIs thinks about this quite a bit.  Even i think
about related  utility functions every now and then (even if it more
from a directed evolution perspective, and probably for about 1% of the
time that you spend on it)

>   Remember that thinking about this problem 
> class requires realizing that superintelligence is an important domain.  An 
> AI that doesn't have the power to tile the universe with tiny smiley faces, 
> that can only produce smiley faces by making humans smile, will appear to 
> work fine during its childhood.  And thinking also requires realizing that 
> the problem is difficult and urgent enough to require thinking through in 
> advance, and so on.  And it requires the ability think about possible 
> disasters without someone needing to nudge you into it, without going 
> mystical, without looking for handy rationalizations why it *might* not 
> happen, and so on.  My experience suggests that this is a bar that is not 
> only above most AI researchers, it is a bar astronomically above most AI 
> researchers.  They never even get close to addressing the difficult and 
> interesting challenge; they fail on the earliest possible part of the problem.
> 
Not sure about the above.  Have you done a comprehensive survey of AI
researchers?  Reading what they publish on journals won't tell you what
they haven't thought about, just what they thought that was fundable.
Lots of smart cookies at universities around the world.  

> So it is not that Earth is inevitably doomed.  The problems are 
> individually solvable, and I even think that the meta-problem is solvable, 
> so that missing one thing won't kill us.  But Earth is doomed by default, 
> and very few people care, or are interested in rising to the challenge.
> 
> Incidentally, Dubrovsky, since you replied with math, which is a very rare 
> thing, can I ask you to start listing other things that you think might go 
> wrong with a Really Powerful Optimization Process?  To see if any of them 
> are problems I missed?
> 

>From what i read in http://www.sl4.org/bin/wiki.pl?CollectiveVolition, i
don't have many concerns with the finished product (i'm not convinced on
why collective instead of individual, but i'm not concerned).  My main
concern is that you can't get There from Here.  The initial state for
the RPOP that takes collective volition into account and has the
property of a suitable, provable invariant is not humanly codable, only
>H codable, which is a bit of a bitch.  It is a bar so high that you
have zero probability of winning the race, and therefore it renders the
whole project irrelevant.  (Of course, i'm saying all this without
knowing what is actually required to get there, but i doubt very much
that you do either.   The impossibly hard bit of "there" i'm referring
to is a formalised version of the initial invariant, not just the
incredibly hard problem of possitive feedback increases in intelligence
while keeping the invariant through code rewrites.).
It's also a pity that you dismiss all other projects as useless. >H
non-feedbacked  narrow(ish) AI (eg Really Good theorem provers) could
make the development of a suitable RPOP much safer (or even at all
obtainable).  Cutting deaths from 50+ million / year down to fuck all /
year also does not require such a huge jump in intelligence.  Yes, the
bar for a script kiddie to destroy the world is always lowering, but
that has been pretty low for a while in the biochem field and still here
we stand.  (btw, an AI to deal with bio threats sounds more immediately
urgent and 10^9 times easier (i don't mean any uber-beast, just a
program that creates instant molecular solutions to viri and a suitable
distribution program))

alejandro