[ExI] Singletons

[Anders Sandberg]

On 2011-01-03 19:10, Eugen Leitl wrote:
> On Mon, Jan 03, 2011 at 06:30:06PM +0100, Anders Sandberg wrote:
>
>> Which could be acceptable if the rules are acceptable. Imagine that
>
> The question is who makes the rules? Imagine a lowest common
> denominator rule enforcer, using quorum of all people on this
> planet. A very scary thought.

Not entirely different from Eliezer's Coherent Extrapolated Volition. 
Although the idea is to be a bit more sophisticated than "lowest common 
denominator" (and this is of course where things rapidly become complex 
and interesting to philosophers, but tricky to implement).

The singleton design problem and the friendly AI problem seem to be 
similar, maybe even identical. We want to define a structure that can be 
relied on to not misbehave even when expanded beyond the horizons we 
know when we design it. Singletons might not have to be 
superintelligent, although that is likely a desirable property of a 
singleton.

My own favored answer to the friendly AI problem is that since the 
design part looks very hard and we know we can make reasonable stable 
and self constraining communities of minds (they are called societies), 
we should aim at that instead. But this presupposes that the "hard 
takeoff in a basement somewhere" scenario is unlikely. If we have reason 
to think that it might matter then we better get the friendliness 
working before it happens, prevent AI emergence or direct it towards 
safer forms. Similarly for singletons, if we think there is unlikely to 
be any threats worth the risk of singletons we can just let things 
coordinate themselves. But if we think there are serious threats around, 
then we better figure out how to make singletons, prevent the 
singleton-worthy threats somehow else, or make the threats 
non-singleton-worthy. In any case, figuring out how to figure out 
upcoming xrisks well seem to be a good idea.


> Aargh. So the singleton can do whatever it wants by tweaking the
> physical layer.

I think that is the standard singleton. Scary enough, but then there is 
the motivational singleton (can control the motivations of all agents) 
and identity singleton (it is all agents). Controlling the physical 
substrate might be less powerful than controlling motivations.


-- 
Anders Sandberg
Future of Humanity Institute
Oxford University