[ExI] Yes, the Singularity is the greatest threat to humanity

[Anders Sandberg]

Stefano Vaj wrote:
>
> I am still persuaded that the crux of the matter remains a less 
> superficial consideration of concept such as "intelligence" or 
> "friendliness".  I suspect that at any level of computing power, 
> "motivation" would only emerge if a deliberate effort is made to 
> emulate human (or at least biological) evolutionary artifacts such as 
> sense of identity, survival istinct, etc., which would be certainly 
> interesting, albeit probably much less crucial to their performances 
> and flexibility than one may think.

"Motivation" does not have to be anything like human motivations. As 
Wikipedia says, "Motivation is the driving force which causes us to 
achieve goals." - a chess playing system can be said to have a 
motivation to win games built into itself, just like Schmidthuber's 
Gödel machine and Hutter's AIXI have a motivation to maximize their 
utility functions. Now, the sub-goals that emerge in the later two AI 
architectures seem likely to be utterly alien to human motivations for 
nearly any utility function.

Whether we would get instrumental subgoals corresponding to Omohundro AI 
drives remains to be seen. His (and others) arguments are qualitative 
and not formalized; I think it would be a really good project to try to 
prove, disprove them or delimit under what circumstances they occur.


>
> Now, are we really in the business of transhumanism to advocate for 
> the enforcement of a global, public control of tech progress in the 
> field of information technology aimed at slowing down its already 
> glacial pace? I think there are already more than enough people who 
> are only too happy to preach for the adoption of such measures...

Actually thinking about the risks and problems before promoting 
technologies is a sane thing. If there is a big danger with it we better 
think about effective solutions to it. I'm rather a transluddite than 
promoter of every shiny new technology - cobalt bombs are shiny too.

-- 
Anders Sandberg,
Future of Humanity Institute
Philosophy Faculty of Oxford University