[extropy-chat] Re: Bad Bayesian - no biscuit! (was A New Year's gift for Bayesians)

[Eliezer Yudkowsky]

Brett Paatsch wrote:
 >
 >     "Imagine that you wake up one morning and your left arm
 >      has been replaced by a blue tentacle. The blue tentacle
 >      obeys your motor commands - you can use it to pick up
 >      glasses, drive a car, etc. How would you explain this
 >      hypothetical scenario? Take a moment to ponder this
 >      puzzle before continuing."
 >
 > So I did imagine it. I imagined it in good faith, and I imagined it
 > consistent with a spirit of exploration and good will built that Eliezer
 > had established through the early part of his essay.

What about the spirit of cunning plots and mischief?  For the life of me, I 
don't understand why people are so shocked to find that I have a goofy 
sense of humor; it's not as if my writings don't provide ample warning. 
Anyway...

I'm trying to forge rationality into a new and more coherent art, reusing 
my l33t build-a-mind-from-scratch skillz to go beyond that accumulated 
handicraft of rationality passed down from generation to generation. 
Sometimes I may say things that shock a listener raised in the ancient 
tradition.

 > Where Eliezer had placed "spoiler space", I stopped reading and I
 > wrote down my explanation. (I'd been reading with pen in hand and
 > making critical notes in the margin.)

Got a scanner?  I wouldn't mind seeing a copy of the notes (if that 
wouldn't be too much effort for you).

 > It seemed to be fair and
 > scientific to provide an answer *before* reading on so as not to
 > contaminate the experiment. )
 >
 > I wrote (and I quote):
 >
 >     "I'd "explain" it provisionally as some surprising organisation
 >     of people had entered my house and replaced my arm whilst
 >     I slept with technology I didn't know existed.
 >
 >     I'd be bewildered. Frightened even. But I'd not think "magic"
 >     had occurred".

I also wouldn't think "magic" had occurred.  The magic hypothesis is rather 
vague without further specification, has tended to be disproved to the 
extent it has been historically specified, and makes no particular 
prediction of my arm being replaced with a tentacle.

 > And then, with the heightened curiosity of one who has escalated
 > their commitment I went back to see what Eliezer the Bayesian,
 > Eliezer the spreader-of-analogical-probability-clay-mass would have
 > done.
 >
 > And he'd written this.
 >
 >    "How would I explain the event of my left arm being replaced
 >     by a blue tentacle? The answer is that I wouldn't. It isn't going
 >     to happen."
 >
 > Email perhaps can't convey my exact reaction to that but here's
 > the comments I wrote in the margin.
 >
 > ----
 >
 > "No. No. No. You cheated Eliezer. You cheated!
 >
 > You can't assign a probability of zero!*
 >
 > Not fair!! You said it did happen. You're being dishonest with the
 > data to say it's not going to happen."

Yep, that's a traditional response.  :P

I was hoping for pretty much that reaction from my readers, who I assumed 
would be traditional rationalists.

The ideal of traditional rationality is that reality is allowed to tell you 
anything it wants, and you ought to shut up and listen - a stance arising 
from the sad human tendency to deny experimental evidence when it conflicts 
with something more valuable, like hope or authority.

What's less commonly appreciated is that reality does *not* tell you just 
anything.  Reality is *extremely constricted* in what it tells you, far 
more constricted than human storytelling.  Truth is *not* stranger than 
fiction; humans just have a warped idea of what constitutes normality.  The 
theory of conservation of momentum is not that momentum is conserved *most* 
of the time or *nearly all* of the time, it is that momentum is conserved 
*every single time*.  It isn't a coincidence, we hypothesize, that 
conservation of momentum has been observed on every occasion thus yet where 
humanity has had opportunity to test it; it's because reality obeys this 
*absolute* rule, and this absolute rule has given rise to humanity's 
experience so far.

Fallible humans may not dare to assign a probability of unity to the 
probability of conservation of momentum.  Let us say we assign a 
probability of at most 90%, meaning that if we took ten contemporary 
physical hypotheses of equal status, we would expect at least one to be 
disconfirmed a millennium hence.  But the hypothesis of conservation of 
momentum is not that momentum is conserved 90% of the time or even 99.9999% 
of the time.  The hypothesis of conservation of momentum is that momentum 
is conserved 100.00000% of the time.  We may be uncertain, but the 
hypothesis of "conservation of momentum" hypothesizes a state of affairs in 
which reality is *not* uncertain; a reality in which it is *absolutely 
certain* that momentum will be conserved on each and every occasion.

The absolute character of physical law is foreign to human thinking.  When 
humans make rules, there's always some unwritten give-and-take, some room 
for compromise.  Even some atheists are shocked by the uncompromising 
attitude of radical foaming-at-the-mouth atheist hard-liners like myself, 
that there is *no* room for magic in the universe, that there is *no* 
separate magisterium reserved for thoughts "spiritual" or "religious" or 
"magical".  And to those who claim that every belief system partakes of the 
transcedent, I say:  Yes, every known culture divides the universe into a 
mundane realm of verifiable assertions and a sacred realm where sloppy 
thinking is allowed; but I'm saying that's a mistake.

It is not a coincidence that every sacred hypothesis in human history which 
infringes on testability has been falsified.  (And that is nearly all 
sacred hypotheses.  The makers of such hypotheses lack the skill in 
rationality to make the hypotheses genuinely unfalsifiable.  If they 
understood what constitutes Bayesian evidence, they wouldn't be 
constructing sacred hypotheses.)  The most probable cause of humanity's 
experience so far is that we live in an *absolutely* mundane universe, 
mundane in every place and every time with 100.00000000000000% frequency.

By asserting that conservation of momentum is *absolute*, the hypothesis 
sticks out its neck very far, and perhaps its neck will be sliced off by an 
experiment that violates conservation of momentum - but that's part of the 
point.  If someone reports an experiment that violates conservation of 
momentum, you shouldn't chalk it down to a rare exception to the general 
rule (maybe someone negotiated the laws of physics down a little from their 
extreme and unreasonable position that momentum should be conserved on 
every single occasion).

Experiments in cognitive psychology show that people fear air travel more 
than car travel.  Why?  Because airplane accidents are more frequently 
reported in the media.  People's fear of a particular hazard is, generally, 
directly proportional to how often that hazard is reported in the media. 
Rare hazards that kill lots of people, like airplane crashes, tend to be 
reported in the media often; common small hazards go relatively unnoticed. 
  This bias is only one of many experiments where people have been observed 
reason, not from the actual statistical frequencies of cases, but from the 
most attention-grabbing cases or most commonly reported cases.  I think 
that part of the Way is trying to fit yourself to the real world, to the 
actual statistical frequency of events - deliberately thinking about the 
statistical likelihood of any sample case presented for your attention.

No one in all human history has ever woken up with a functioning tentacle 
in place of their arm.  You should have noticed that when I asked you to 
find an explanation for it.  Yes, you were tricked.  But it's a kind of 
trickery that goes on all the time and which I wish my readers to notice. 
For example, a thought experiment which postulates "zombies", people that 
behave exactly like people in this universe except that they are not 
conscious.  (Why do they write papers on consciousness, including 
discussion of zombie experiments, if they are not conscious?  For more on 
this see Dennett's "The Unimaginable Preposterousness of Zombies".)  Or 
consider Damien Broderick's swift reaction to John C. Wright's attempt to 
quote Scrooge as an argument against skepticism.  Damien didn't actually 
use my phrase "logical fallacy of generalization from fictional evidence", 
but I hope my writing had something to do with it.

Occasionally I tread on the futile task of trying to persuade people not to 
buy lottery tickets, and they say something along the line of "Someone has 
to win!" or "You can't win if you don't play!"  To which the answer is, 
"'Someone' will not be you.  You will not win the lottery, period.  I could 
make a hundred thousand statements of equal strength and not be wrong even 
once.  I have *godlike* confidence that you will not win the lottery.  Stop 
thinking about the pleasantness of the outcome, stop attending to this 
improbable event, because it will *not happen*."  I ask people to stop 
buying lottery tickets, and they think, "But what if I would have won?", 
and imagine a thought experiment which, though physically possible - not 
*absolutely* impossible like violating conservation of momentum - has a 
probability so low that *they shouldn't be thinking about it while making 
the decision*.

I would advise philosophers who indulge in thought experiments to 
consciously weight the emotional force of a thought experiment by the 
probability of that thought experiment, so that they may learn to live in 
this, our real world.

Lottery players are led astray by attending to hypothetical events of great 
emotional intensity and extremely low probability.  So too, John C. Wright 
was led astray by attending to a powerful and sympathetic experience, 
Scrooge's discovery of magic and the triumphant comeuppance of skepticism, 
which never actually happened.  Nor will it happen, ever, in this our 
mundane universe.

You won't wake up with a blue tentacle in place of an arm.  Remember that, 
the next time someone asks you to imagine the impossible, or even the 
statistically improbable.  You followed where I asked you to go, imagined 
what I asked you to imagine, and then found yourself betrayed.  But next 
time you will notice when someone tries to lead you astray.

Let us learn to live in this universe the way it really is, attending to 
the real frequency of events instead of the frequency of media reports of 
events.  When someone asks us to imagine a magical outcome, let us forget 
all of the novels we have read, and all the movies we have seen, and all 
the hopes of our childhood, and remember that the observed frequency is zero.

At the same time, let's not forget how ridiculous the 20th century would 
have sounded if you'd reported it to a 19th-century listener.  But reality 
is very constrained in what kind of ridiculousness it presents us with. 
Not one of the ridiculous things that happened in the 20th century violated 
conservation of momentum.

Oh... and if you *do* wake up with a tentacle in place of an arm... it's 
probably not because anyone snuck into your room; there must be a simpler, 
more likely explanation you didn't think of, or the event wouldn't have 
happened.  When you have only a poor explanation, one that doesn't make 
things ordinary in retrospect, just admit you don't have an explanation, 
and keep going.  Poor explanations very, very rarely turn out to be 
actually correct.  A gang of people sneaking into your room with unknown 
technology is a poor explanation.  Whatever the real explanation was, it 
wouldn't be that.  If that's the best you can do, then "I wasn't expecting 
this and I have no clue why it happened or what will happen next" is a far 
superior answer.  If I demanded that you produce a specific hypothesis 
anyway, you should have told me to stuff it.  Why develop a habit of 
producing hypotheses that can't be right?  What good will it serve you?

In real life, sometimes we don't know what happened.  Real life is where we 
will actually apply our l33t rationality skillz, so "I don't know" was the 
correct answer, taking the impossible experience as a fixed given.  If I'd 
sworn that I really did possess some concrete reason to anticipate that you 
might *actually* wake up with a tentacle, and asked you to guess my good 
reason, "I don't know" would have been the correct answer (taking my 
rationality as a fixed given).

 > Most of what I know of Bayesian reasoning I know as a result of
 > reading Eliezer's two essays on it.

If you enjoy the art, read more literature on it.  "Rational Choice in an 
Uncertain World" is good.  If you're ready for academic stuff, start with 
"Judgment Under Uncertainty".

 > So perhaps if my understanding
 > of Bayesian reasoning or inference is wrong I can escape by
 > blaming Eliezer for it :-)

Whoa, I'm a highly atypical Bayesian.  If you're judging Bayesian academia 
by me, that's, er, not a good idea.  I did warn my readers that Technical 
Explanation would be controversial.

 > I suspect, on the basis of those two essays that I am a Bayesian
 > although I didn't know I was and so I haven't been calling myself
 > one. The merit I see in the Bayesian approach is that it manages
 > uncertainties more carefully and consistently then most people do
 > intuitively. [And boy does the world need that].

Human beings aren't designed as Bayesians (though our thoughts sometimes 
have Bayes-structure, which is why we work at all).  The word "Bayesian" 
usually refers, not to someone who actually implements Bayesian principles 
(for that is humanly unattainable), but to someone who espouses Bayesian 
principles.  I don't think you could be a Bayesian without knowing it, 
unless you had unwittingly demanded that people be principled about 
assigning prior probabilities, or some such stance which today is commonly 
known as "Bayesian".

 > So, it's 2005.  I'm a Bayesian. And as long as I'm wearing metaphorical
 > teeshirts I'm also a Bright.
 >
 > Regards,
 > Brett Paatsch
 >
 > *  Eliezer's assigning a probability of zero to observed facts however
 > unlikely those facts might have been a priori is the reason for my
 > heading this post Bad Bayesian - no buscuit.

You can't assign a probability of 0.000000..., but you could theoretically 
assign a probability of 1e-100 if you think you can get away with it, 
because it isn't actually *zero* - the logarithm is minus a thousand 
decibels, but not actually negative infinity.  1e-100 is effectively zero, 
but there's a world of difference between effectively zero and actually 
zero.  My point is that you should be wary of probabilities which are, 
*given* the dominant physical hypothesis, actually zero or effectively zero.

If your explanation:  "A secret organization of people entered my house and 
replaced my arm with a tentacle using unknown technology" doesn't make you 
anticipate (even just a little) waking up with a tentacle tomorrow in this 
our real world, then it's a poor explanation.  For it is this, our real 
world, in which you must live.  Nor should you bother trying to develop a 
better explanation.  For in this, our real world, you will have no need of it.

-- 
Eliezer S. Yudkowsky                          http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence