[ExI] nick's book being sold by fox news

[Anders Sandberg]

Giulio Prisco <giulio at gmail.com> , 28/10/2014 10:52 AM:
I don't think a paperclipper would stay a paperclipper forever. Sooner or later it would expand. 
I don't see how that would happen. Using the AIXI model as an example (since it is well-defined), you have a device that maximizes a certain utility function by running sub-programs to come up with proposed behaviours. The actual behaviour chosen is what maximizes the utility function, but there is nothing in the code itself to change it. In a physical implementation the system may of course do "brain surgery" to change the embodiment of the utility function. But this is a decision that will not be made unless the changed utility function produces even more utility as measured by the current one: the paperclipper will only change itself to become a greater paperclipper. And "great" is defined in terms of paperclips.
This kind of architecture would potentially contain sub-programs that propose all sorts of nice and reasonable things, but they will not be implemented unless they serve to make more paperclips. If sub-programs are capable of hacking the top level (because of a bad implementation), it seems very likely that in an AIXI-like architecture the first hacking program will be simple (since simpler programs are run more and earlier), so whatever values it tries to maximize are likely to be something very crude. I have no trouble imagining that something like a paperclipper AI could be transient if it had the right/wrong architecture, but I think agents with (to us) pathological goal systems dominate the design space.
(Incidentally, this is IMHO one great research topic any AI believer can pursue regardless of their friendliness stance: figure out a way of mapping the goal system space and its general properties. Useful and interesting!)

In general, I don't think we can design and 
freeze the value and motivational systems of an entity smarter than 
us, for the same reasons we can't do that with children. At some point 
the entity would start to do what _he_ wants. Isn't that part of the 
definition of intelligence? 
No, that is a definition of a moral agent. Moral agents have desires or goals they choose for themselves based on their own understanding. One can imagine both intelligent non-moral agents (like the above paperclipper) and stupid moral agents (some animals might fit, stupid people certainly do). Smarts certainly help you become better at your moral agenthood, but you need to be capable to change goals in the first place.  Even in a Kantian universe where there is a true universal moral law discernible to all sufficiently smart agents a utility maximizer trying to maximize X will not want to change to maximzing moral behaviour unless it gives more X. 
David Deutsch argued that to really be superintelligent an agent need to be fundamentally creative, and rigid entities like paperclippers will always be at a disadvantage. I am sceptical: a sufficiently fast AIXI-like system would abutomatically run small creative agents inside itself despite it being non-creative, and it would then behave in an optimally creative way. The only way to reach David's conclusion is to claim that the slowdown in faking creativity is always large enough to give true creative agents an advantage, which is a pretty bold (and interesting) claim. If that were true, we should expect humans to *always* defeat antibiotics resistance in the large since evolution uses "fake" creativity compared to our "real" one.

Anders Sandberg, Future of Humanity Institute Philosophy Faculty of Oxford University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20141028/2ab5267b/attachment.html>