[ExI] Future of Humanity Institute at Oxford University £1 million grant for AI

Giulio Prisco giulio at gmail.com
Sun Jul 5 14:38:05 UTC 2015


Anders, the paperclip maximizer is a good example indeed. For all we
know, it could reason that humans are the only users of paperclips in
the known universe, so that humans give a cosmic meaning to paper clip
production, so that humans must be revered and protected... My point
is that we don't know - just like Columbus didn't know what he would
find at the end of his journey.

On Sat, Jul 4, 2015 at 10:28 AM, Anders Sandberg <anders at aleph.se> wrote:
> Från: Giulio Prisco <giulio at gmail.com>
>
> For superAIs, remember that using the analogy in Nick's book we are
> talking of _really_ smarter entities, not in the sense that Einstein
> is smarter than the village idiot, but in the sense that humans are
> smarter then beetles. Beetles couldn't control humans for long - they
> couldn't lock me in a room, because they aren't smart enough to have
> locks and keys. Etc.
>
>
> Don't get me wrong, I am super happy that the FHI got the funding
> because you and Nick are my friends and I am sure the FHI will do
> something good with the money, but I still think that hoping to
> influence, control, condition, program superAIs is a  contradiction in
> terms.
>
>
> We both agree that superintelligent systems are fundamentally uncontrollable
> once they are up and running. But that does not mean one cannot start them
> in different states *before* they become powerful, and some of these states
> are much safer than others. It is just that right now we do not have a good
> theory of how to tell (that is part of the whole research program). However,
> we do have models of super-AI that we know are deeply unsafe and actually
> likely to misbehave despite being arbitrarily smart (my standard example is
> the AIXI-driven paperclip maximizer - it is well defined enough that you can
> prove stuff about it).
>
> If one thinks there is nothing one can plan for when making smarter AIs,
> then it just becomes a gamble with existential risk. Sensible people would
> likely want to avoid AI then, while people with overconfident metaethical
> views  would push forward (I have met a few). If one thinks AI development
> is going to be slow enough or have economies of scale that produces a broad
> swell rather than a spike, fine, different AI systems can act as checks and
> balances on each other and there is a good chance humanity is integrated in
> the cooperative framework... but how do we know that scenario is the one
> that will play out? It is a good idea to understand AI dynamics as well as
> we can before we bet *everything* on our understanding.
>
> There is an interesting interplay between views of knowability and
> controllability of the future here. A lot of traditional AI people think the
> future is knowable and controllable, and hence safe ("Look, I am simply not
> going to make my robot want to harm people, right?"). That is often an
> overconfident position when talking about things further down the road. Then
> there are those who think the future isn't knowable but controllable ("If
> the AI misbehaves we will just pull the plug"). That seems historically to
> have often been a bad assumption (let's just stop emitting CO2, right?).
> Thinking the future isn't controllable or knowable is just a fatalistic
> "whatever will be, will be" - it doesn't motivate anybody to anything. The
> uncontrollable but knowable corner is even worse: this is where people think
> they know what will happen but there is no way of avoiding it. As I see it,
> moving things towards controllability is generally a good thing: it cannot
> always be done, but it is good to know what can be done. We can also push
> towards knowing more, which hopefully both allows better aiming of whatever
> control there is, and to counteract overconfidence about the field.
>
>
>
>
> G.
>
>
>
> On Fri, Jul 3, 2015 at 6:57 PM, Anders Sandberg <anders at aleph.se> wrote:
>> Från: Giulio Prisco <giulio at gmail.com>
>>
>> They should have sent a couple of hundred bucks my way, and I would
>> have advised them to leave the rest of the money in the bank.
>> Superintelligent AIs will do what they want to do. That's the
>> definition of intelligence, super or not. Trying to program or enforce
>> behaviors or values in a super-smart AI is like telling your smart and
>> rebellious kids to stay home and study instead of going out and have
>> fun. Same thing, and same result.
>>
>>
>> But the current approach to AI safety is like never talking with the kids
>> about morals, emotions or societal conventions, nor giving them feedback
>> on
>> what they do except instrumental success ("Great work in forcing open the
>> gun cabinet!") What we aim at doing is rather like figuring out what kind
>> of
>> upbringing is less likely to produce school shootings, sociopathy or
>> unhappy
>> career choices.
>>
>> Also, there are the lesser AIs to be concerned about. You want to make
>> sure
>> they can interpret our intentions, laws or norms in ways that actually
>> works. Superintelligent entities may be smart enough to be safe even when
>> merely "smart" agents are very unsafe (but see the whole analysis of why
>> emergent AI values are not guaranteed to stay close to ours or anything
>> sane; Inceptionist pictures are a pretty good example of what happens when
>> we let AI preferences run free
>>
>> http://d.ibtimes.co.uk/en/full/1445360/psychedelic-images-generated-by-googles-neural-network.jpg
>> )
>>
>>
>> Anders Sandberg, Future of Humanity Institute Philosophy Faculty of Oxford
>> University
>>
>>
>> _______________________________________________
>> extropy-chat mailing list
>> extropy-chat at lists.extropy.org
>> http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat
>>
>
> _______________________________________________
> extropy-chat mailing list
> extropy-chat at lists.extropy.org
> http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat
>
>
>
>
>
> Anders Sandberg, Future of Humanity Institute Philosophy Faculty of Oxford
> University
>
>
>
> _______________________________________________
> extropy-chat mailing list
> extropy-chat at lists.extropy.org
> http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat
>




More information about the extropy-chat mailing list