[ExI] Singletons

[Eugen Leitl]

On Wed, Jan 05, 2011 at 11:30:47AM +0100, Anders Sandberg wrote:

> Not entirely different from Eliezer's Coherent Extrapolated Volition.  

I have yet to see something in CEV worth criticizing. So far, it's
a lot of vague handwaving.

> Although the idea is to be a bit more sophisticated than "lowest common  
> denominator" (and this is of course where things rapidly become complex  
> and interesting to philosophers, but tricky to implement).

As soon as things start becoming complex, we're pretty close to
human design complexity ceiling. So you need to build a critical
seed for hard takeoff (in order to create the initial capability
assymmetry, and hence enforcement ability), yet you can't validate 
anything much about what exactly will happen almost immediately
after. Danger, danger Will Robinson.

> The singleton design problem and the friendly AI problem seem to be  
> similar, maybe even identical. We want to define a structure that can be  

I thought they were the very same exact idential thing.

> relied on to not misbehave even when expanded beyond the horizons we  
> know when we design it. Singletons might not have to be  
> superintelligent, although that is likely a desirable property of a  

Singletons will be pretty much to be superintelligent, because 
they will be under constant, heavy attack by everyone with a spare
neuron (unless said spare neurons are condidered to be too dangerous,
and universal lobotomy is mandated along with arrest to self-enhancement
through Darwinian evolution. Isn't it nice to be be almost omniscient
and almost omnipotent?)

> singleton.
>
> My own favored answer to the friendly AI problem is that since the  
> design part looks very hard and we know we can make reasonable stable  
> and self constraining communities of minds (they are called societies),  
> we should aim at that instead. But this presupposes that the "hard  

Yeah, pretty much so.

> takeoff in a basement somewhere" scenario is unlikely. If we have reason  

Basement is unlikely, unless it's the basement of a massively black
funded military project, which is attempting to preempt other said
project. I do not think anything like that exists, though of course
it would be difficult to identify, other than by tracing publications,
monetary streams, and particular purchases.

> to think that it might matter then we better get the friendliness  
> working before it happens, prevent AI emergence or direct it towards  

In order to get friendliness, you must first define friendliness.
Then build an evolution constrainer, asserting conservation of that
metric. 

> safer forms. Similarly for singletons, if we think there is unlikely to  
> be any threats worth the risk of singletons we can just let things  
> coordinate themselves. But if we think there are serious threats around,  

I think we should let things happen, while executing oversight about
anything which could produce Blight. Like said military black projects.

> then we better figure out how to make singletons, prevent the  
> singleton-worthy threats somehow else, or make the threats  
> non-singleton-worthy. In any case, figuring out how to figure out  
> upcoming xrisks well seem to be a good idea.

We do not seem to be making much progress in that area in the last
20-30 years. Admittedly, almost nobody is working on it, but it might
be it's a very hard problem.

>
>> Aargh. So the singleton can do whatever it wants by tweaking the
>> physical layer.
>
> I think that is the standard singleton. Scary enough, but then there is  
> the motivational singleton (can control the motivations of all agents)  

If you can tweak the physical layer, you can tweak the motivations of
all agents, since they're all operating at the same physical layer.
This can be a very subtle effect, which cumulates over time. The
eventual result is a twisted, warped, evil thing.

> and identity singleton (it is all agents). Controlling the physical  
> substrate might be less powerful than controlling motivations.

All computation is embodied, motivation is just a particular 
computation. Consider how parasites influence behaviour, the
easiest way to implement the cop is the cop infesting your CNS,
or a controlling nanoparasite network in every head, along with
support infrastructure everywhere. 

It would, of course, induce selective autoagnosia in all its
hosts.

Sounds friendly enough yet? But we haven't even started.

-- 
Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org
______________________________________________________________
ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE