[ExI] Organizations to "Speed Up" Creation of AGI?

[Anders Sandberg]

On 2011-12-25 18:04, Stefano Vaj wrote:
> An AGI is simply a (presumably fast, unless we want wait for eons at
> each interaction) computer, executing a program that emulates well
> enough human features to perform as well as an actual human in a
> Turing test, or in some other tasks where current computers are
> currently very poor.

However, an AGI is a general problem solver that will act to maximize 
the utility it can gather. This can lead to instrumental drives that 
might be very problematic:
http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf
(this paper is a great star, but needs to be formalized and analyzed 
better!)

Humans are "safe" because we are such a shoddy implementation of general 
problem solving and utility maximization. We do not have pure top level 
goals, we do not adjust ourselves or out environments very rationally to 
achieve our goals.


-- 
Anders Sandberg
Future of Humanity Institute
Oxford University