[ExI] Self improvement

[Richard Loosemore]

Keith Henson wrote:
> I suspect, for speed of light reasons, that there will be a lot of AIs
> rather than one.
> 
> If there are, I sure hope they will be friendly toward each other.  It
> would be just awful to be caught in a war waged by AIs.

But... this is where just I cannot understand the standard line on 
"whether an AI would be friendly".

The question of whether the AIs would be friendly toward each other is a 
matter of design, very much the same as the question of whether or not 
they would be good at arithmetic.

You wouldn't say "I sure hope they will be good at artithmetic", you 
would say "I sure hope somebody will remember to include a good 
arithmetic module, or a good set of arithmetic concepts".  And, if you 
said that, everyone would kind of yawn and say "Yeah well, we hope the 
designers paint it a nice color too, but so what?"

Now, you can question the accuracy of the analogy -- is it really as 
easy to design the behavior as it is to insert an arithmetic module? -- 
but no matter how strongly you dispute my analogy, what you can't do is 
blatantly, without supporting argument, talk as if they diametric 
opposite is the case.  You can't say "I wonder whether they will be 
friendly..." or "I sure hope they will be friendly..." as if the 
friendliness is SELF-EVIDENTLY a 100% emergent property that we have no 
control over.

The [assumed] lack of control over their motivational behavior is not 
self-evident:  you have to show some argument as to why we have no say 
over the motivation.  And to do that, you have to speak to specific 
motivational designs, and not just assume that all motivational designs 
are the same.

I bring this up frequently.  The challenge is never answered (only 
ignored, or deflected into some other discussion).


Richard Loosemore


P.S.

[I say "you" of course, not meaning you personally, but the point of 
view you represented there.  That point of view is widespread].