[ExI] How could you ever support an AGI?

[Richard Loosemore]

Henrique Moraes Machado wrote:
> Richard Loosemore> Unfortunately, I think I was not clear enough, and as a 
> result you have
>> misunderstood what I said in rather a substantial way.
>> When you build an AGI, you *must* sort out the motivation mechanism
>> ahead of time, or the machine will simply not work at all.  You don't
>> build an AGI and *then* discover what its motivation is.
> 
> Let me throw more wood into the fire then.
> I am not afraid of AGIs at all. But let's imagine a scenario where some 
> humans begin to worship the AGI as some sort of deity (yes, humans can be 
> that messed up). A sect could quickly evolve around the AGI and THAT is 
> something I consider dangerous. Not the AGI itself, but those possible post 
> modern religious fanatics. The difference between these new fanatics and the 
> present day fanatics that we all know and don't love is that their god 
> really exists (I'm sure there's already some distopic scifi novel that has 
> already addressed this issue). 

I think there is a difficulty here, which has to do with the context in 
which you assume an AGI to exist.

Without going into details, the context is what 100% determines whether 
this scenario (a) could happen, (b) is likely to happen, and (c) would 
be a problem even if it did happen.

The reason I say this is that I have thought about the future timeline 
of an AGI arrival, and I think that the overwhelming majority of 
possible futures involve situations in which this AGI-worship scenario 
would not be a problem.  Because of my analysis, therefore, it is hard 
to answer your question.

I think this kind of problem (where different people make wildly 
different assumptions about the context in which an AGI would arise) is 
at the root of many discussions and misunderstandings.

I suppose what that means is that we have to spend more time examining 
possible-futures scenarios.



Richard Loosemore