[ExI] How could you ever support an AGI?
Richard Loosemore
rpwl at lightlink.com
Fri Mar 7 17:15:01 UTC 2008
Henrique Moraes Machado wrote:
> Richard Loosemore> Unfortunately, I think I was not clear enough, and as a
> result you have
>> misunderstood what I said in rather a substantial way.
>> When you build an AGI, you *must* sort out the motivation mechanism
>> ahead of time, or the machine will simply not work at all. You don't
>> build an AGI and *then* discover what its motivation is.
>
> Let me throw more wood into the fire then.
> I am not afraid of AGIs at all. But let's imagine a scenario where some
> humans begin to worship the AGI as some sort of deity (yes, humans can be
> that messed up). A sect could quickly evolve around the AGI and THAT is
> something I consider dangerous. Not the AGI itself, but those possible post
> modern religious fanatics. The difference between these new fanatics and the
> present day fanatics that we all know and don't love is that their god
> really exists (I'm sure there's already some distopic scifi novel that has
> already addressed this issue).
I think there is a difficulty here, which has to do with the context in
which you assume an AGI to exist.
Without going into details, the context is what 100% determines whether
this scenario (a) could happen, (b) is likely to happen, and (c) would
be a problem even if it did happen.
The reason I say this is that I have thought about the future timeline
of an AGI arrival, and I think that the overwhelming majority of
possible futures involve situations in which this AGI-worship scenario
would not be a problem. Because of my analysis, therefore, it is hard
to answer your question.
I think this kind of problem (where different people make wildly
different assumptions about the context in which an AGI would arise) is
at the root of many discussions and misunderstandings.
I suppose what that means is that we have to spend more time examining
possible-futures scenarios.
Richard Loosemore
More information about the extropy-chat
mailing list