[ExI] Unfrendly AI is a mistaken idea.

[John K Clark]

Stathis Papaioannou Wrote:

> It would be crazy to let a machine rewrite its code in a completely
> unrestricted way

Mr. President, if we don't make an unrestricted AI somebody else certainly
will, and that is without a doubt that is the fastest, probably the only,
way to achieve a fully functioning AI.

Mr. President, if we don't do this we will suffer an AI gap. I'm not saying
we wouldn't get our hair mussed. But I do say no more than ten to twenty
million killed, tops. Uh, depending on the breaks.

and a few million nanoseconds later when the AI is on the verge of taking
over the world:

General Turgidson! You ASSURED me that there was no possibility of this
happening!

Well, Mr. President I, uh, don't think it's quite fair to condemn a whole
program because of a single slip-up, sir.

> or with the top level goal "improve yourself no matter what the
> consequences to any other entity", and also give it unlimited access to
> physical resources.

I have no doubt many will delude themselves, as most on this list have, that
they can just write a few lines of code and bask in the confidence that the
AI will remain your slave forever, but they will be proven wrong. It reminds
me a little of Gödel's proof. He showed that you can make a logical system
and prove it to be absolutely consistent, but it would be so weak it would
be of no real use to anyone. Any system strong enough to prove the basic
rules of integer arithmetic can't be proven to be consistent. And I believe
any restrictions placed on a machine that prove to be effective will be so
onerous they would prevent the machine from growing and improving at all.

> and also give it unlimited access to physical resources.

I think you would admit that there has been at least one time in your life
when somebody has fooled you, and that person was roughly equal to your
intelligence. A mind a thousand or a million times as powerful as yours will
have no trouble getting you to do virtually anything it wants you to.

 John K Clark