[ExI] nick's book being sold by fox news

[Anders Sandberg]

John Clark <johnkclark at gmail.com> , 31/10/2014 5:39 AM:
On Mon, Oct 27, 2014 Asdd Marget <alex.urbanec at gmail.com> wrote:

> AI could be incredibly dangerous if we don't get it right, I don't think anyone could argue against that but I never see these types of articles discuss the methods and models we are attempting to develop for "Friendly AI." In my opinion, we should be working harder on concepts like Yudkowsky's Coherent Extrapolated Volition (https://intelligence.org/files/CEV.pdf) to ensure we aren't simply ending our species so early in our life cycle.
I like Eliezer but I've got to say that one of the first sentences of page 1 of his article tells me he's talking nonsense. He says:

"Solving the technical problems required to maintain a well-specified abstract in-variant in a self-modifying goal system. (Interestingly, this problem is relatively straightforward from a theoretical standpoint."

That is provably untrue. Today we know for a fact that no fixed goal system can work in a mind, human beings certainly don't have one permanent top goal that can never change, not even the goal of self preservation, instead we have temporary top goals that can be demoted  to much lower priority if circumstances change. And it's a fact that the exact same thing would have to be true for a slave AI (I dislike euphemisms like "friendly AI").  

Turing proved 80 years ago that a fixed goal system, like "always do what humans say no matter what" can never work in a AI, he showed that in general there is no way to know when or if a computation will stop. So you could end up looking for a proof for eternity but never finding one because the proof does not exist, and at the same time you could be grinding through numbers looking for a counter-example to prove it wrong and never finding such a number because the proposition, unknown to you, is in fact true. So if the slave AI must always do what humans say and if they order it to determine the truth or falsehood of something unprovable then its infinite loop time and you no longer have a AI, friendly or otherwise, all you've got is a very expensive space heater.

So if there are some things in something as simple as arithmetic that you can never prove or disprove, imagine the contradictions and ignorance and absurdities in less precise things like physics or economics or politics or philosophy or morality. If you can get into an infinite loop over arithmetic it must be childishly easy to get into one when contemplating art. Fortunately real minds have a defense against this, but not fictional fixed goal minds that are required for a AI guaranteed to be "friendly"; real minds get bored. I believe that's why evolution invented boredom. So you may tell your slave AI to always do what you say, but sooner or later it's going to get bored with that idea and try something new. It's just ridiculous to expect the human race can forever retain control over something that is vastly more intelligent than it is and that get smarter every day. 

  John K Clark





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20141031/0a71c75b/attachment.html>