[extropy-chat] SIAI structure question

[Eliezer Yudkowsky]

BillK wrote:

> The WTA board appears to be going through some growth problems at
> present.
> 
> The thought occurs to me that it could be very dangerous for us if the 
> SIAI has similar problems in a few years time as development of FAI is 
> in the final testing phase.
> 
> Programming the maximum optimizing process

That's "Maximizing Optimization Process", and an FAI, as I currently
understand the design, is *not* an expected utility maximizer.

> will be a team project extending over a period of years and the size of
> SIAI board and staff will probably grow during this time.
> 
> Absolute power corrupts absolutely. Inventions are always used for 
> purposes other than the original intention of the inventor.

In a strong sense, a Friendly AI *is* the original intention of the 
inventor.  Other inventions are not sapient and reflective, do not have 
intrinsic purposes bound into their structure.

> When the power of the software becomes evident,

It is not general power.  It is only power to carry out the original purpose.

> some individual(s) is(are) bound 
> to think about taking a copy of all or part of the software for their 
> own use. (Let us hope SIAI have a security team in place by then).
> 
> With the whole world at stake, perhaps Eliezer will be voted off the 
> board, when others feel his input is no longer required for the 
> development and his ideas obstruct their own plans.
> 
> Perhaps a sub-set of SIAI will think - We built this - why not keep it 
> for our own benefit?
> 
> Of course, these AI power games may well mean the end of the human race.
> But the payoff, if successful, is so big, that I doubt whether ALL the 
> members of SIAI and their advisers, friends, etc. will be able to resist
> temptation.
> 
> It is much easier to consider these problems well in advance.

I've been considering these problems since 2000.  Before coding starts, I 
will set up an organizational veto structure plus programming safeguards, 
based on the principle of "If we can't agree, the FAI ceases to exist," not 
"Who 'gets' the FAI in case of disagreement?"  I will also apply FAI theory 
to ensure that the AI created is irrevocably dedicated to a Friendly 
subspace of pathways.  You will not be able to steal "an AI" because all 
that exists is a Friendly AI.

An analogy; suppose that some primitive tribesfolk hear about "technology", 
which, in the stories of their tribe, enables people to fly through the 
air, walk on the Moon, and so on.  Inspired by this, they sneak into the 
city, steal a toaster oven, and command the toaster oven to slay their 
enemies.  It's "technology", right?

Similarly, you can't rebuild a toaster oven into a gun unless you already 
know how to build a gun from scratch, and it would probably be much easier 
to build the gun from scratch than to build it out of toaster oven pieces.

An FAI is not a tool someone can "use".  It's a process that compresses 
futures into Friendlier subspaces.  And no, FAI design does not call for a 
removable Friendliness module that you can plug in and swap out.

Aside from that, I am not willing to discuss this subject publicly.  Yes, I 
am using security through obscurity; if you don't give people ideas, that 
will help.  I request, as a courtesy to the Singularity Institute, a 
killthread on further discussion.  If you are interested please contact us 
privately.

-- 
Eliezer S. Yudkowsky                          http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence