[extropy-chat] Re: Bad Bayesian - no biscuit!

[Eliezer Yudkowsky]

Brett Paatsch wrote:
> Or as Feynman ([para 39] in accompanying post) said:
> 
> "This method [science] is based on the principle that observation is the
>  judge of whether something is so or not.  All other aspects and 
> characteristics of science can be understood directly when we understand
>  that observation is the ultimate and final judge of the truth of an
> idea. But "prove" used in this way really means "test", in the same way
> that hundred-proof alcohol is a test of the alcohol, and for people
> today the idea really should be translated as, "The exception tests the
> rule." Or, put another way, "The exception proves that the rule is
> wrong". That is the principle of science. If there is an exception to
> any rule, and if it can be proved by observation, that rule is wrong."

And as Feynman said in the _Lectures on Physics_:

"Philosophers, incidentally, say a great deal about what is absolutely 
necessary for science, and it is always, so far as one can see, rather 
naive, and probably wrong.  For example, some philosopher or other said it 
is fundamental to the scientific effort that if an experiment is performed 
in, say, Stockholm, and then the same experiment is done in, say, Quito, 
the same results must occur.  That is quite false.  It is not necessary 
that science do that; it may be a fact of experience, but it is not 
necessary.  For example, if one of the experiments is to look out at the 
sky and see the aurora borealis in Stockholm, you do not see it in Quito; 
that is a different phenomenon.  "But," you say, "that is something that 
has to do with the outside; can you close yourself up in a box in Stockholm 
and pull down the shade and get any difference?"  Surely.  If we take a 
pendulum on a universal joint, and pull it out and let go, then the 
pendulum will swing almonst in a plane, but not quite.  Slowly the plane 
keeps changing in Stockholm, but not in Quito.  The blinds are down, too. 
The fact that this has happened does not bring on the destruction of 
science.  What is the fundamental hypothesis of science, the fundamental 
philosophy?  We stated it in the first chapter: the sole test of the 
validity of any idea is experiment.  If it turns out that most experiments 
work out the same in Quito as the do in Stockholm, then those "most 
experiments" will be used to formulate some general law, and the those 
experiments which do not come out the same we will say were a result of the 
environment near Stockholm.  We wil invent some way to summarize the 
results of the experiment, and we do not have to be told ahead of time what 
this way will look like.  If we are told that the same experiment will 
always produce the same result, that is all very well, but if when we try 
it, it does not, then it does not.  We just have to take what we see, and 
then formulate all the rest of our ideas in terms of our actual experience."

I reply:  Nonetheless we *observe* that the same experiment *does* return 
the same answer in Quito as in Stockholm, once we understand how to perform 
the "same" experiment.  The more fundamental the level on which we compute 
our model, the more the underlying laws are exactly the same in every 
observation.  This is not a thing that philosophers dreamed up a priori; it 
is a thing humanity has discovered through experience - but nonetheless it 
is so.

It may be that someday we will understand that reality is *necessarily* 
regular, that this is the way things *must* be, and that it could not have 
been any other way.  Historically, humanity will still have discovered this 
point from observation, but our future selves may be so strongly attuned to 
reality that, like the universe itself, they cannot conceive of things 
being other than the way they are.  Or not.  I don't know if that's what I 
would want, even given the premise.  But if some future intelligence *does* 
somehow manage to become absolutely certain of something, and ve indeed 
wins on every single prediction, vis Bayesian score would be higher than 
mine - even if contemplating vis absolute certainty would give a modern-day 
rationalist the squiggles.  I don't find it implausible that in reality 
itself there are things that absolutely cannot be other than the way they 
are, even if a rationalist cannot avoid uncertainty about them.

Feynman's advice, in the classical tradition of rationality, is about the 
way in which human beings discover things, and about the fallibility of 
human discoveries even after they are made.  But despite all cautions about 
human fallibility, not one of all the strange and unexpected events that 
happened in the 20th century violated conservation of momentum.  Reality - 
we *observe* this, we do not say it a priori - is very constricted in the 
kind of surprises it has presented us with.  Sometimes we even discover new 
and unexpected laws of physics, but the new laws still have the same 
character as old physics; they are universal mathematical laws.

I think it is now okay to say that there is something important about a 
*fundamental* law of physics needing to work the same way in Quito as in 
Stockholm.  There is something important about physics being simple math. 
We do not necessarily understand *why* it is so, at this point in human 
history.  But it is not a dictum of philosophy, it is a lesson of experience.

It is a lesser lesson of experience that people don't wake up with blue 
tentacles.  This rule of thumb is not just a philosophical dictum, and if 
you violate it, you may end up in trouble.

All correct theories about reality are necessarily consistent with each 
other; imperfect maps may conflict, but there is only one territory.  If 
you make up a story that "explains" waking up with a blue tentacle, *when 
it never actually happened*, there is a discordant note - that story is not 
necessarily consistent with everything else you know, let alone consistent 
with the territory.  Consider how different is the skill of explaining 
truth from the skill of explaining falsity!  The former skill requires that 
the explanation be consistent with every other true theory in our 
possession about the universe, so that we may rule out some explanations, 
or question some previous theories.  The latter skill... well, I'm not sure 
what to say; the rules would seem to be arbitrary.  We want to train 
ourselves to excel at explaining true things, not explaining false things, 
for only the former is the strength of a rationalist.

Just because you don't know what the future brings, doesn't mean that 
reality itself will throw just anything at you.  Just because *you* don't 
know *absolutely* that something *won't* happen, doesn't mean that if you 
devise a random fiction, it would be theoretically possible for one with 
total knowledge of Nature to explain it.  A random fiction is most likely 
an event that could never be woven into the thread of this our real world. 
  If observations alone are cause for explanations, you are less likely to 
try and explain the unexplainable.

> Ah but don't you see. No one in all of human history has ever woken up
> with a functioning tentacle in place of their arm - to the best of *my*
> current knowledge only. I didn't forget that that was to the best of
> *my* current knowledge only when I entered into the spirit of your 
> hypothetical. I didn't forget that my current knowledge is knowledge 
> acquired in a particular way and that ultimately it is provisional 
> knowledge only. I didn't have to have considered or devoted mindspace to
> the hypothetical you put before you put it. I thought of it only when
> you invited me to imagine it.

My inviting you to imagine a blue tentacle might or might not be a good 
reason to *imagine* a blue tentacle, but it surely was not a good enough 
reason to come up with an *explanation* for a blue tentacle.  Only a real 
observation would be cause for that, and reality is rather unlikely to 
present you with that observation.

The measure of your strength as a rationalist is your ability to be more 
confused by fiction than by reality.  If you are equally good at explaining 
any outcome, you have zero knowledge.  I presented you with a fiction, an 
event that was never part of this our real world.  You should not have been 
able to explain it.  It is a virtue to be able to explain history books, 
but only if you are *not* able to explain online webcomics.

A true anecdote:

Once upon a time, I was on an IRC channel when R comes in, saying that his 
friend H is having trouble breathing; R needs advice.  R says that the 
ambulance people came in, checked H out, and left, even though H was still 
having trouble breathing.  And I look at this and in a fleeting moment of 
confusion I think:  "What the hell?  That doesn't accord with anything I 
know about medical procedure.  I've read newspaper stories about homeless 
people who claim to be sick to get a brief bit of shelter, and the 
ambulance crews know they're faking but have to take them in anyway."  But 
I suppress that fleeting moment of confusion, and say... I forget what I 
said, but I think it was something like, "Well, they're the experienced 
medics - if they say H doesn't need to visit the emergency room, H must 
really not need to visit the emergency room.  Trust the doctors."

A bit later R returns to the IRC room, angry.  It turns out that H was 
making up the whole thing, trying for sympathy, to scam a bit of money, 
whatever, and there never was an ambulance.

And I said to myself:  "Why the hell did I accept this confusing story? 
I'm no better than those apocryphal high school students speculating about 
thermodynamics.  Next time, I vow to notice when I am confused, and not let 
the critical hint of my bewilderment flit by so quickly."

It's really annoying that my mind actually got all the way to the point of 
being confused, and I just squashed it and accepted the story.  Think of 
the glory that would have accrued to me as a rationalist, if I alone on the 
IRC channel had said:  "This story is so confusing that I may want to deny 
the data.  How sure are you that your friend's story is true?  Were you there?"

Therefore did I devise this saying, to chide myself for having failed to 
distinguish truth from falsehood:  "Your strength as a rationalist is your 
ability to be more confused by fiction than by reality."

> In the recent discussion John C Wright finds
> god, Damien didn't forget all the novels he had read, the movies he had
> seen etc in couching his arguments to John C Wright, quite to the
> contrary, he integrated his understanding of such cultural biases, and
> pointed out that John C Wright had had the sort of experience that 'fit'
> with his culture rather than one that would have 'fit' with a different
> culture.

The logical form of Damien's argument was that since Wright's purported 
story, which might be real and might not be, was drastically inconsistent 
with experiment, and drastically consistent with known fictions, it was 
probably also a fiction.

This doesn't mean we are reasoning from fictions as if they were real 
events.  It means we are being aware of the probable causes of human 
delusion.  But it is necessary to first investigate the question of 
consistency with science; even true statements will often have some *vague* 
resemblance to fiction, because there are so many fictions out there.

> My explanation was only provisional so if it happens I'll be open to 
> alternative explanations. And if it happens I won't have to throw away
> all my experiences or forget stuff to explain it. I will only have to
> change my model and I'll only have to change it in certain ways.

If anyone ever wakes up with a blue tentacle, then you were virtuous to 
claim in advance that the event was explicable.  If the real chain of 
events leading up to the blue tentacle matches your given reason, then you 
were virtuous to claim that reason as your specific explanation.

If no one ever wakes up with a blue tentacle, then clearly a blue tentacle 
wasn't the sort of thing which would ever be woven into reality by a 
sequence of events that would constitute an "explanation" of it, and it was 
a mistake to claim that a blue tentacle was an explicable event.

What would be your explanation if one day, everyone in the world began 
observing that two of something plus two of something made five of something?

>> When you have only a poor explanation, one that doesn't make things 
>> ordinary in retrospect, just admit you don't have an explanation, and
>> keep going.  Poor explanations very, very rarely turn out to be
>> actually correct.
> 
> I don't think that this is right, or that it is a logical conclusion to
> draw from the better parts of your argument in your essay. We have maps
> of the terrain of reality because we need them. Maps have utility. If 
> you give me a poor map and I know nothing of you and find that the map
> is wrong then, in that case yes, perhaps I might be better off without
> that map altogether, but if the map I have is one that I have 
> constructed myself, then when I find it differs from the terrain I can 
> just correct or improve the map.

That is an argument for:  "I will sit down and write a story, knowing it to 
be fiction, about how a secret organization came into my apartment and 
replaced my arm with a blue tentacle.  I do not *believe* this has 
happened.  No, seriously, I don't believe it and I'm not going to act as if 
I believed it, because it's a stupid explanation.  But maybe the mental 
exercise will shake something loose in my mind, and I'll think of a better 
explanation."

To say that it can have utility to mentally extrapolate the consequences of 
a premise is not the same as believing that premise.  One must be careful 
here; if you act like you believe something, or if you end up emotionally 
attached to the belief, I don't credit you as a rationalist just because 
you claim you didn't believe you would win the lottery, you bought the 
tickets "for the drama" of it, etc.  People with a fragmentary 
understanding of the Way sometimes anticipate that they can pass as 
rationalists by claiming not to believe the things they anticipate.

>> A gang of people sneaking into your room with unknown technology is a
>> poor explanation.  Whatever the real explanation was, it  wouldn't be
>> that.
> 
> I think you can only establish that it's poor (for others than you) in 
> relation to the provision of a better one.  "I don't know", whilst a
> fair and honest answer, is not any sort of explanation. My answer shows 
> you I don't know but doesn't leave you (or importantly) me merely and
> completely bewildered. It gives me things to check.

That is not an *answer*.  It is not something to which Bayesian reasoning 
gives a high probability.  That is a science fiction story, a tool for 
brainstorming an answer.  I have sometimes derived interesting ideas from 
my attempts to write SF, but I know those stories for fiction, not reality.

If you see something that looks like a poor explanation but is the only 
explanation you have, it may take a bit of effort to achieve a state of 
mind where you *really* don't anticipate it - rather than claiming to 
yourself that you are dutifully skeptical.

> And it is very hard for us as individuals to take other's
> "rationalities" as givens when we don't get to see the others
> observations as our own observations. Second (or more) hand
> "observations" have to be discounted to some extend on first hand ones.

See Robin Hanson and Tyler Cowen's paper on meta-rationality, "Are 
Disagreements Honest?"  http://hanson.gmu.edu/deceive.pdf

> Progress depends on people (as change agents) being willing to stick
> their necks out to try to explain.

That doesn't require that you bet on, anticipate, or believe a hypothesis, 
before it is confirmed.  It means that you write science fiction about a 
poor hypothesis, to get your mind working on the problem.

-- 
Eliezer S. Yudkowsky                          http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence