[ExI] Organizations to "Speed Up" the Creation of AGI?

[Anders Sandberg]

Longish post. Summary: soft takeoffs have a good chance of being nice 
for us, hard ones might require some hard choices. I give reasons for 
why I think we might be in the range 0.1-1% risk of global disaster per 
year. I urge a great deal of caution and intellectual humility.

On 2011-12-20 06:35, Kevin G Haskell wrote:
> While the concern is valid, how would FHI and like-minded groups go
> about ensuring that once AGI is created, either in 10 years or 200, that
> this new species will be anything other than it/they want(s) to be, and
> do to whatever existing species, rather human or Transhuman, that will
> still have much lower levels of speed, awareness, and power, that it wants?

(That sentence structure is worthy Immanuel Kant :-) )

There is a tricky theoretical ethical question surrounding just what 
kind of ethical agents they would be, and how we could recognize it. But 
from a practical ethics perspective, I can see a bunch of possibilities:

If there is no hard takeoff, we should expect a distribution of "power" 
that is fairly broad: there will be entities of different levels of 
capability, and groups of entities can constrain each others activities. 
This is how we currently handle our societies, with laws, police, 
markets, and customs to constrain individuals and groups to behave 
themselves. Our solutions might not be perfect, but it doesn't stretch 
credulity too much to imagine that there are equivalents that could work 
here too.

(Property rights might or might not help here, by the way. I don't know 
the current status of the analysis, but Nick did a sketch of how an AGI 
transition with property rights might lead to a state where the *AGIs* 
end up impoverished even if afforded full moral rights. More research is 
needed!)

A problem might be if certain entities (like AGI or upload clades) have 
an easy way of coordinating and gaining economies of scale in their 
power. If this is possible (good research question!!!), then it must 
either be prevented using concerted constraints from everybody else or a 
singleton, or the coordinated group better be seeded with a few entities 
with humanitarian values. Same thing if we get a weakly multilateral 
singularity with just a few entities on par.

In the case of hard takeoffs we get one entity that can more or less do 
what it wants. This is likely very bad for the rights or survival for 
anything else unless the entity happens to be exceedingly nice. We are 
not optimistic about this being a natural state, so policies to increase 
the likelihood are good to aim for. To compound the problem, there might 
be incentives to have a race towards takeoff that disregards safety. One 
approach might be to get more coordination among the pre-takeoff powers, 
so that they 1) do not skimp on friendliness, 2) have less incentives to 
rush. The result would then be somewhat similar to the oligopoly case 
above.

Nick has argued that it might be beneficial to aim for a singleton, a 
top coordinating agency whose will *will* be done (whether a 
sufficiently competent world government or Colossus the Computer) - this 
might be what is necessary to avoid certain kinds of existential risks. 
But of course, singletons are scary xrisk threats on their own...

As I often argue, any way of shedding light on whether hard or soft 
takeoffs are likely (or possible in the first place) would be *very 
important*. Not just as cool research, but to drive other research and 
policy.


> I would be interested in how you can quantify the existential risks as
> being 1% per year? How can one quantify existential risks that are
> known, and as yet unknown, to mankind, within the next second, never
> mind the next year, and never mind with a given percentage?

For a fun case where the probability of a large set of existential risks 
that includes totally unknown cosmic disasters can be bounded, see 
http://arxiv.org/abs/astro-ph/0512204

My own guesstimate is based on looking at nuclear war risks. At least in 
the Cuba crisis case some estimates put the chance of an exchange to 
"one in three". Over the span of the 66 years we have had nuclear 
weapons there have been several close calls - not just the Cuba Crisis, 
but things like Able Archer, the Norwegian rocket incident, the NORAD 
false alarms 79/80 etc. A proper analysis needs to take variable levels 
of tension into account, as well as a possible anthropic bias (me being 
here emailing about it precludes a big nuclear war in the recent past) - 
I have a working paper on this I ought to work on. But "one in three" 
for one incident per 66 years gives a risk per year of 0.5%. (Using 
Laplace's rule of succession gives a risk of 0.15% per year, by the way) 
We might quibble about how existential the risk of a nuclear war might 
be, since after all it might just kill a few hundred million people and 
wreck the global infrastructure, but I give enough credence to the 
recent climate models of nuclear winter to think it has a chance of 
killing off the vast majority of humans.

I am working on heavy tail distributions of wars, democides, pandemics 
and stuff like that; one can extrapolate the known distributions to get 
estimates of tail risks. Loosely speaking it all seems to add to 
something below 1% per year.

Note that I come from a Bayesian perspective: probabilities are 
statements about ignorance, they are not things that exist independently 
in nature.



> As someone who considers himself a Transhumanist, I come to exactly the
> opposite conclusion as the one you gave, in that I think by focusing on
> health technologies and uploading as fast as possible, we give humanity,
> and universal intelligence, a greater possibility of lasting longer as a
> species, being 'superior' before the creation of AGI,and perhaps merging
> with a new species that we create which will 'allow' us to perpetually
> evolve with it/them, or least protect us from most existential threats
> that are already plentiful.

I personally do think uploading is the way to go, and should be 
accelerated. It is just that the arguments in favor of it reducing the 
risks are not that much stronger than the arguments it increases the 
risks. We spent a month analyzing this question, and it was deeply 
annoying to realize how uncertain the rational position seems to be.


> Once a brain is emulated, a process that companies like IBM have
> promised to complete in 10 years because of competitive concerns, not to
> mention all of the other companies and countries pouring massive amounts
> of money for the same reason, the probability that various companies and
> countries are also pouring ever larger sums of money into developing
> AGI,  especially since many of the technologies overlap.  If
> brain-emulation is achieved in 10 years or less, then AGI can't be far
> behind.

Ah, you believe in marketing. I have a bridge to sell you cheaply... :-)

As a computational neuroscientist following the field, I would bet 
rather strongly against any promise of brain emulation beyond the insect 
level over the next decade. (My own median estimate ends up annoyingly 
close to Kurzweil's estimate for the 2040s... )

Do you have a source on how much money countries are pouring into AGI? 
(not just narrow AI)


> Still, I can't really see how waiting for brain-emulation will somehow
> keep us safer as a species once AGI is actually developed. What factors
> are being used in the numbers game that you mentioned?

Here is a simple game: what probability do you assign to us surviving 
the transition to an AGI world? Call it P1. Once in this world, where we 
have (by assumption) non-malign very smart AGI, what is the probability 
we will survive the invention of brain emulation? Call it P2.

Now consider a world where brain emulation comes first. What is the 
chance of surviving that transition? Call it P3. OK, we survived the 
upload transition. Now we invent AGI. What is the chance of surviving it 
in this world? Call it P4.

Which is largest, P1*P2 or P3*P4? The first is the chance of a happy 
ending for the AGI first world, the second is the chance of a happy 
ending for the uploading first world.

Now, over at FHI most of us tended to assume the existence of nice 
superintelligence would make P2 pretty big - it would help us avoid 
making a mess of the upload transition. But uploads doesn't seem to help 
much with fixing P4, since they are not superintelligent per se (there 
is just a lot more brain power in that world).



> What is the general thinking about why we need to wait for full-brain
> emulation before we can start uploading our brains (and hopefully
> bodies)?  Even if we must wait, is the idea that if we can create
> artificial brains that are patterned on each of our individual brains,
> so that we can have a precise upload, that the AGIans will somehow have
> a different view about what they will choose to do with a fully
> Transhumanist species?

I don't think you would be satisfied with a chatbot based on your online 
writing or even spoken speech patterns, right?

You shouldn't try to upload your brain before we have full-brain 
emulation since the methods are likely going to be 1) destructive, 2) 
have to throw away information during processing due to storage 
constraints until at least mid-century, 3) we will not have evidence it 
works before it actually works. Of course, some of us might have no 
choice because we are frozen in liquid nitrogen...


>  >>(My own strategy is to talk to as many AI researchers as possible and
>  >>get them thinking in constructive ways. Stopping research has never been
>  >>an option, but it might get smarter.)
>
> When you said 'constructive' and 'smarter,' don't you mean 'slower and
> more cautious?'  I don't mean to put words in your mouth, but I don't
> see what else you could mean.

I tell them about their great forebears like Simon, Minsky and McCarthy, 
and how they honestly believed they would achieve human level and beyond 
AI within their own active research careers. Then I point out that none 
of them - or anybody else for that matter - seemed to have had *any* 
safety concerns about the project. Despite (or perhaps because of) 
fictional safety concerns *predating* the field.

I point out that if they take their own ideas seriously they should also 
be concerned about getting the consequences right. They cannot 
simultaneously claim they are pursuing AGI and that 1) it will change 
the world radically and 2) it will automatically be safe, unless they 
have some fairly strong theory of AI safety and idea why it will be 
implemented. And in that case, they better tell others about it.

Another thing I suggest is that they chat with philosophers more. OK, 
that might seriously slow down anybody :-) But it is surprising how many 
scientists do elementary methodological, ethical or epistemological 
mistakes about their own research - discussing what you do with a 
friendly philosopher can be quite constructive (and might bring the 
philosopher a bit more in tune with real research).


> May I ask if you've been polling these researchers, or have a general
> idea as to what the percentages of them working on AGI think regarding
> the four options I presented (expecting, of course, that since they are
> working on the creation of them, few are likely in support of either the
> stop, or reversing options, but rather the other two choices of go
> slower or speed up)?

I have not done any polling like that, but we did do a survey at an AI 
conference we arranged a year ago:
http://www.fhi.ox.ac.uk/news/2011/?a=21516

Fairly optimistic about AGI soonish (2060), concerned with the 
consequences (unlikely to be just business as usual), all over the place 
in regards to methodology, and cautious about whether Watson would win 
(the survey was done before the win).

-- 
Anders Sandberg
Future of Humanity Institute
Oxford University