[ExI] nick's book being sold by fox news

[Mike Dougherty]

On Tue, Oct 28, 2014 at 10:54 AM, Anders Sandberg <anders at aleph.se> wrote:
> This kind of architecture would potentially contain sub-programs that
> propose all sorts of nice and reasonable things, but they will not be
> implemented unless they serve to make more paperclips. If sub-programs are
> capable of hacking the top level (because of a bad implementation), it seems
> very likely that in an AIXI-like architecture the first hacking program will
> be simple (since simpler programs are run more and earlier), so whatever
> values it tries to maximize are likely to be something very crude. I have no
> trouble imagining that something like a paperclipper AI could be transient
> if it had the right/wrong architecture, but I think agents with (to us)
> pathological goal systems dominate the design space.

If "Core Wars" was any model for the design space, the "copy yourself
to a random location in the core, execute there" is a pretty simple
and fast replicator.  That it would frequently destroy larger programs
by stomping into/over their instruction space was a secondary effect.