[ExI] Future of Humanity Institute at Oxford University £1 million grant for AI

Anders Sandberg anders at aleph.se
Sat Jul 4 08:28:19 UTC 2015

Från:   Giulio Prisco <giulio at gmail.com> 
For superAIs, remember that using the analogy in Nick's book we are
talking of _really_ smarter entities, not in the sense that Einstein
is smarter than the village idiot, but in the sense that humans are
smarter then beetles. Beetles couldn't control humans for long - they
couldn't lock me in a room, because they aren't smart enough to have
locks and keys. Etc.

Don't get me wrong, I am super happy that the FHI got the funding
because you and Nick are my friends and I am sure the FHI will do
something good with the money, but I still think that hoping to
influence, control, condition, program superAIs is a  contradiction in
We both agree that superintelligent systems are fundamentally uncontrollable once they are up and running. But that does not mean one cannot start them in different states *before* they become powerful, and some of these states are much safer than others. It is just that right now we do not have a good theory of how to tell (that is part of the whole research program). However, we do have models of super-AI that we know are deeply unsafe and actually likely to misbehave despite being arbitrarily smart (my standard example is the AIXI-driven paperclip maximizer - it is well defined enough that you can prove stuff about it).

If one thinks there is nothing one can plan for when making smarter AIs, then it just becomes a gamble with existential risk. Sensible people would likely want to avoid AI then, while people with overconfident metaethical views  would push forward (I have met a few). If one thinks AI development is going to be slow enough or have economies of scale that produces a broad swell rather than a spike, fine, different AI systems can act as checks and balances on each other and there is a good chance humanity is integrated in the cooperative framework... but how do we know that scenario is the one that will play out? It is a good idea to understand AI dynamics as well as we can before we bet *everything* on our understanding.

There is an interesting interplay between views of knowability and controllability of the future here. A lot of traditional AI people think the future is knowable and controllable, and hence safe ("Look, I am simply not going to make my robot want to harm people, right?"). That is often an overconfident position when talking about things further down the road. Then there are those who think the future isn't knowable but controllable ("If the AI misbehaves we will just pull the plug"). That seems historically to have often been a bad assumption (let's just stop emitting CO2, right?). Thinking the future isn't controllable or knowable is just a fatalistic "whatever will be, will be" - it doesn't motivate anybody to anything. The uncontrollable but knowable corner is even worse: this is where people think they know what will happen but there is no way of avoiding it. As I see it, moving things towards controllability is generally a good thing: it cannot always be done, but it is good to know what can be done. We can also push towards knowing more, which hopefully both allows better aiming of whatever control there is, and to counteract overconfidence about the field. 


On Fri, Jul 3, 2015 at 6:57 PM, Anders Sandberg <anders at aleph.se> wrote:
> Från: Giulio Prisco <giulio at gmail.com>
> They should have sent a couple of hundred bucks my way, and I would
> have advised them to leave the rest of the money in the bank.
> Superintelligent AIs will do what they want to do. That's the
> definition of intelligence, super or not. Trying to program or enforce
> behaviors or values in a super-smart AI is like telling your smart and
> rebellious kids to stay home and study instead of going out and have
> fun. Same thing, and same result.
> But the current approach to AI safety is like never talking with the kids
> about morals, emotions or societal conventions, nor giving them feedback on
> what they do except instrumental success ("Great work in forcing open the
> gun cabinet!") What we aim at doing is rather like figuring out what kind of
> upbringing is less likely to produce school shootings, sociopathy or unhappy
> career choices.
> Also, there are the lesser AIs to be concerned about. You want to make sure
> they can interpret our intentions, laws or norms in ways that actually
> works. Superintelligent entities may be smart enough to be safe even when
> merely "smart" agents are very unsafe (but see the whole analysis of why
> emergent AI values are not guaranteed to stay close to ours or anything
> sane; Inceptionist pictures are a pretty good example of what happens when
> we let AI preferences run free
> http://d.ibtimes.co.uk/en/full/1445360/psychedelic-images-generated-by-googles-neural-network.jpg
> )
> Anders Sandberg, Future of Humanity Institute Philosophy Faculty of Oxford
> University
> _______________________________________________
> extropy-chat mailing list
> extropy-chat at lists.extropy.org
> http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat

extropy-chat mailing list
extropy-chat at lists.extropy.org

Anders Sandberg, Future of Humanity Institute Philosophy Faculty of Oxford University

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20150704/d8fac4a7/attachment.html>

More information about the extropy-chat mailing list