[ExI] Survival

[Stathis Papaioannou]

On 02/01/2008, Jef Allbright <jef at jefallbright.net> wrote:

> On 1/1/08, Stathis Papaioannou <stathisp at gmail.com> wrote:
> > On 02/01/2008, Eliezer S. Yudkowsky <sentience at pobox.com> wrote:
> > > Stathis Papaioannou wrote:
> > > > If you
> > > > read everything I have ever written that has appeared on the Internet,
> > > > neither you nor a Jupiter brain will be able to deduce whether I
> > > > prefer chocolate or vanilla ice cream.
> > >
> > > You prefer vanilla.
> >
> > If you're trying to get me to confirm or deny that, it won't work.
>
> FWIW, you're broadcasting a pretty clear "vanilla" signal to me too.

The best you can do is come up with a probability for vanilla. The
best a superintelligent AI can do is come up with a more accurate
value for that probability than you can. The AI won't  say Pr(vanilla)
= 1 or Pr(vanilla) = 0 because that would just be wrong: there isn't
enough information to make such a prediction.

For ice cream preference there is a right answer and it would be
relatively easy to obtain it with a high degree of certainty by, for
example, talking to people who know me; a human of average
intelligence who has the opportunity to do that will beat an AI which
doesn't every time. When it comes to persuading me to do something I
wouldn't normally do it isn't at all clear that any speech will work.
Even if there is a speech that will work there is no guarantee that an
AI will have enough information about me to figure out what it is. The
AI may conclude that speech A is most likely to make me perform X,
with Pr = 0.1. A human who has more information may conclude that
speech B is most likely to make me perform X, with Pr = 0.2. An AI
with perfect information running a simulation of my brain may conclude
that speech C is the one, with Pr = 0.3, making allowances for the
effects of classical chaos and/or quantum indeterminacy.




-- 
Stathis Papaioannou