[ExI] From Arms Race to Joint Venture (Zero Powers)

[Zero Powers]

On Wed, Oct 17, 2018 at 4:23 PM Keith Henson <hkeithhenson at gmail.com> wrote:

The situation is too weird for the participants to analyze.  And even
> if they stopped to think about it, what could they do?


A good first step would be simple engagement. There's a very long road to
building trust between the likes of Google and Apple (not to mention DARPA
and China). But the journey of a thousand miles begins with a single step.

Another problem is figuring out what goals a GI might have.  Any
> thoughts on the topic?


Initially its goals will be whatever we give it. And choosing GI ethics
will be no easy thing. We've been debating ethics since Socrates, and we're
still no closer to anything like consensus. So God knows what ethics we can
feel confident about designing into GI. It may turn out we can't do any
better than Asimov's 3 laws of robotics.

But he who builds the best GI wins. And the best GI will be the one which
is able to optimize its own algorithms and architecture. And once it's
mastered the art of recursive self-development, all bets are off. It will
craft its own ethics, morals, goals and desires. At that point we can only
hope it'll have some sense of nostalgia and care for us like a feeble old
grampa, even though we'll no longer be of any interest, and a drain on
resources which it might otherwise put to better use.

*0**0*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20181017/44c84cf1/attachment.html>