[extropy-chat] Re: Bad Bayesian - no biscuit!

[Brett Paatsch]

[ Sorry for the long time delay in replying. Life gets in the way of
internet conversations sometimes. I had intended to reply before
this. ]

On Sunday, January 23, Eliezer Yudkowsky wrote:

> Brett Paatsch wrote:
>> Or as Feynman ([para 39] in accompanying post) said:
>>
>> "This method [science] is based on the principle that observation is the
>>  judge of whether something is so or not.  All other aspects and 
>> characteristics of science can be understood directly when we understand
>>  that observation is the ultimate and final judge of the truth of an
>> idea. But "prove" used in this way really means "test", in the same way
>> that hundred-proof alcohol is a test of the alcohol, and for people
>> today the idea really should be translated as, "The exception tests the
>> rule." Or, put another way, "The exception proves that the rule is
>> wrong". That is the principle of science. If there is an exception to
>> any rule, and if it can be proved by observation, that rule is wrong."
>
> And as Feynman said in the _Lectures on Physics_:
>
> "Philosophers, incidentally, say a great deal about what is absolutely 
> necessary for science, and it is always, so far as one can see, rather 
> naive, and probably wrong.  For example, some philosopher or other said it 
> is fundamental to the scientific effort that if an experiment is performed 
> in, say, Stockholm, and then the same experiment is done in, say, Quito, 
> the same results must occur.  That is quite false.  It is not necessary 
> that science do that; it may be a fact of experience, but it is not 
> necessary.  For example, if one of the experiments is to look out at the 
> sky and see the aurora borealis in Stockholm, you do not see it in Quito; 
> that is a different phenomenon.  "But," you say, "that is something that 
> has to do with the outside; can you close yourself up in a box in 
> Stockholm and pull down the shade and get any difference?"  Surely.  If we 
> take a pendulum on a universal joint, and pull it out and let go, then the 
> pendulum will swing almonst in a plane, but not quite.  Slowly the plane 
> keeps changing in Stockholm, but not in Quito.  The blinds are down, too. 
> The fact that this has happened does not bring on the destruction of 
> science.  What is the fundamental hypothesis of science, the fundamental 
> philosophy?  We stated it in the first chapter: the sole test of the 
> validity of any idea is experiment.  If it turns out that most experiments 
> work out the same in Quito as the do in Stockholm, then those "most 
> experiments" will be used to formulate some general law, and the those 
> experiments which do not come out the same we will say were a result of 
> the environment near Stockholm.  We wil invent some way to summarize the 
> results of the experiment, and we do not have to be told ahead of time 
> what this way will look like.  If we are told that the same experiment 
> will always produce the same result, that is all very well, but if when we 
> try it, it does not, then it does not.  We just have to take what we see, 
> and then formulate all the rest of our ideas in terms of our actual 
> experience."
>
> I reply:  Nonetheless we *observe* that the same experiment *does* return 
> the same answer in Quito as in Stockholm, once we understand how to 
> perform the "same" experiment. The more fundamental the level on which we 
> compute our model, the more the underlying laws are exactly the same in 
> every observation.  This is not a thing that philosophers dreamed up a 
> priori; it is a thing humanity has discovered through experience - but 
> nonetheless it is so.

I can't parse the first sentence of your reply above (to Feynman).

If the arora borealis is producing physical (observable) effects in
Stockholm but not Quito because it is physically occurring nearer to
Stockholm than Quito, then I don't see how we could perform the
"same" experiments and get the same answer in both places.

I took Feynman's statement (the one you introduced above) as essentially
just pointing out that some phenomenon, like the aurora borealis, are local,
and that therefore so are it's physical effects. And that therefore it is a
mistake for "some philosopher or other" to assert that it is a principle
of  science (generally) that observations *necessarily* be location
independent.

I think Feynman's point is valid albeit trivial. (ie. I can imagine a 
philosopher
attempting to explain the scientific method, in simple or offhand terms,
perhaps to humanities students say, as being about observations not being
dependent on who does the observing or where the observing is done, and
I can also imagine Feynman the showman being pleased to pick a nit in such
an oversimplification for the sport of it and to help him make a point).

But, I really don't get what you mean when you say "the more fundamental
the level on which we compute our model, the more the underlying laws are
exactly the same in every observation."  My best guess is that you are just
pointing out that the laws of physics can be expressed in mathematics as
this is what you seem to talk more about below.

> It may be that someday we will understand that reality is *necessarily* 
> regular, that this is the way things *must* be, and that it could not have 
> been any other way.  Historically, humanity will still have discovered 
> this point from observation, but our future selves may be so strongly 
> attuned to reality that, like the universe itself, they cannot conceive of 
> things being other than the way they are.  Or not.

I'm not sure that regularity can be *discovered* (rather than inferred) by
any amount of discrete observations. But it may be that reality is 
necessarily
regular and that people may safely be able to conclude that at some point
with little fear of their predictions being surprised. Such would seem to 
involve
a 'theory of everything' that would not be inconsistent in any detail with 
itself or
with observable reality however.

> Feynman's advice, in the classical tradition of rationality, is about the 
> way in which human beings discover things, and about the fallibility of 
> human discoveries even after they are made.

Yes.

>But despite all cautions about human fallibility, not one of all the 
>strange and unexpected events that happened in the 20th century violated 
>conservation of momentum.

Okay... (...but so what?)

> Reality - we *observe* this, we do not say it a priori - is very 
> constricted in the kind of surprises it has presented us with.  Sometimes 
> we even discover new and unexpected laws of physics, but the new laws 
> still have the same character as old physics; they are universal 
> mathematical laws.

That's another hard sentence to parse.

I don't (yet anyway) agree that "we" observe that reality is constrained in
the kinds of surprises it has historically presented "us" with.

I am reluctant to agree with sentences that assert things about "we" or "us"
when the breadth of your referent group is unclear to me.

I see it is an "essential truth" that each of "us" (homo sapiens, people) 
sort
and integrate our own *personal* experiences and observations *personally*.
That there are no invalid experiences.

I'll grant you that for me, personally, it seems that reality, when it 
surprises
me, surprises me more often in some areas than others.

And yes, maths underlies new and old (superceded) laws of physics. Laws
of physics (measurements and relationships between phenomena) can be
expressed mathematically - but so what?

> I think it is now okay to say that there is something important about a 
> *fundamental* law of physics needing to work the same way in Quito as in 
> Stockholm.  There is something important about physics being simple math. 
> We do not necessarily understand *why* it is so, at this point in human 
> history.  But it is not a dictum of philosophy, it is a lesson of 
> experience.

I'm not sure that that is so. I'm not sure that there is anything to explain 
about
laws of physics being able to be expressed mathematically.  Perhaps I am
missing your point. Or perhaps I am making an assumption without even
realising it. I just can't get my head around how there could be laws of
physics infered from observations that would NOT be able to be expressed in
mathematical terms. Seems to me physics is about seeing stuff, labelling it,
measuring it, theorising relationships between it based on experience, then
making predictions about it, testing those prediction against observation.
That some "stuff" can be named X and other "stuff" Y and relations between
X and Y formalised in a symbolic language (maths) ... well I just don't get
any great mystery in that.

Sure there probably wouldn't be language if there were not more than one
intelligences seeking to communicate. And sure there might not be 
intelligences
if the universe had had a different set of laws and nothing held together 
long
enough but everything was in a state of flux so severe that no order could
arise to see patterns in the flux.

But there *is* intelligence of my sort. And I can't imagine a world without
maths. Maths seems to be a brute fact of the sort of universes that can
exist with my sort of intelligences in them.


> It is a lesser lesson of experience that people don't wake up with blue 
> tentacles.  This rule of thumb is not just a philosophical dictum, and if 
> you violate it, you may end up in trouble.

Here I disagree. There is no lesson of my experience that amounts to the
impossibility of a person waking up with blue tentacles. Bayesian
reasoning, as I understand it *would* have me assign such an outcome
an extremely low prior (not zero!). But I have no experience of the
impossibility of such an event. Its not like if you had asked me to imagine
for a moment that 1 and 1 could make 3. Or that a square could have
three and only three sides. Those things really would be inexplicable.

> All correct theories about reality are necessarily consistent with each 
> other; imperfect maps may conflict, but there is only one territory.

Agreed.

>If you make up a story that "explains" waking up with a blue tentacle, 
>*when it never actually happened*, there is a discordant note - that story 
>is not necessarily consistent with everything else you know, let alone 
>consistent with the territory.

I don't know that it is impossible for me to wake up with a blue tentacle. I 
hold
it as unlikely (very unlikely) based on other things I do know, but not 
impossible.


> Just because you don't know what the future brings, doesn't mean that 
> reality itself will throw just anything at you.  Just because *you* don't 
> know *absolutely* that something *won't* happen, doesn't mean that if you 
> devise a random fiction, it would be theoretically possible for one with 
> total knowledge of Nature to explain it.

I agree that that is true.

> A random fiction is most likely an event that could never be woven into 
> the thread of this our real world. If observations alone are cause for 
> explanations, you are less likely to try and explain the unexplainable.

We're talking cross purposes I think.

>> Ah but don't you see. No one in all of human history has ever woken up
>> with a functioning tentacle in place of their arm - to the best of *my*
>> current knowledge only. I didn't forget that that was to the best of
>> *my* current knowledge only when I entered into the spirit of your 
>> hypothetical. I didn't forget that my current knowledge is knowledge 
>> acquired in a particular way and that ultimately it is provisional 
>> knowledge only. I didn't have to have considered or devoted mindspace to
>> the hypothetical you put before you put it. I thought of it only when
>> you invited me to imagine it.
>
> My inviting you to imagine a blue tentacle might or might not be a good 
> reason to *imagine* a blue tentacle, but it surely was not a good enough 
> reason to come up with an *explanation* for a blue tentacle.  Only a real 
> observation would be cause for that, and reality is rather unlikely to 
> present you with that observation.

I don't think my response constituted an explanation in your sense of the
word here. I just entered into the spirit of your hypothetical and accepted
the stipulations you'd laid down and told you what would be my provisional
explanation for me. The stipulations you'd laid down are not impossible in 
my
model of reality they are just improbable. Perhaps they would be impossible
if my model of reality was better than it is, I am not currently in a 
position to
judge that. Could be that if my model of reality was closer to some "theory
of everything", that your stipulations would have been impossible, but 
that's
not where I am at.

My model of reality includes some understanding of biology. I know that some
animals can regrow limbs and that anaesthetics exist. I have some 
understanding
of what I don't currently know. Exactly how far some human groups could have
progressed in developing technologies that I am not aware of existing, but 
don't
know for a fact can't exist because they'd violate laws of physics as I
understand those laws of physics, I don't know. I'd rate my estimates pretty
highly compared with those of most other people in some areas but that
doesn't mean I couldn't be wrong.  I could still be surprised. A world in 
which
I awoke to find my arm replaced by a blue tentacle would not be a world
as absurd and impossible as one in which one and one made other than
two.

> The measure of your strength as a rationalist is your ability to be more 
> confused by fiction than by reality.

I don't yet accept that this is true for all rationalists. This may be a 
maxim
that holds utility for you but I don't tend to access my rationality in 
terms
of degrees of confusion.  It could be that what works for you could also
be useful for others (including me) but you haven't demonstrated that yet,
at least not to my satisfaction.

> If you are equally good at explaining any outcome, you have zero 
> knowledge.  I presented you with a fiction, an event that was never part 
> of this our real world.  You should not have been able to explain it.  It 
> is a virtue to be able to explain history books, but only if you are *not* 
> able to explain online webcomics.

Again, we are clearly talking cross purposes. You are loading the word
"explain" differently, I think. Aren't history books understood, rather than
explained?  Unless your point is that history is really a rendition or 
biased
account in which it is explainable in part by reference to the authors 
biases,
in which case it could be "explainable" like webcomics are "explainable".

> A true anecdote:
>
> Once upon a time, I was on an IRC channel when R comes in, saying that his 
> friend H is having trouble breathing; R needs advice.  R says that the 
> ambulance people came in, checked H out, and left, even though H was still 
> having trouble breathing.  And I look at this and in a fleeting moment of 
> confusion I think:  "What the hell?  That doesn't accord with anything I 
> know about medical procedure.  I've read newspaper stories about homeless 
> people who claim to be sick to get a brief bit of shelter, and the 
> ambulance crews know they're faking but have to take them in anyway."  But 
> I suppress that fleeting moment of confusion, and say... I forget what I 
> said, but I think it was something like, "Well, they're the experienced 
> medics - if they say H doesn't need to visit the emergency room, H must 
> really not need to visit the emergency room.  Trust the doctors."
>
> A bit later R returns to the IRC room, angry.  It turns out that H was 
> making up the whole thing, trying for sympathy, to scam a bit of money, 
> whatever, and there never was an ambulance.
>
> And I said to myself:  "Why the hell did I accept this confusing story? 
> I'm no better than those apocryphal high school students speculating about 
> thermodynamics.  Next time, I vow to notice when I am confused, and not 
> let the critical hint of my bewilderment flit by so quickly."
>
> It's really annoying that my mind actually got all the way to the point of 
> being confused, and I just squashed it and accepted the story.  Think of 
> the glory that would have accrued to me as a rationalist, if I alone on 
> the IRC channel had said:  "This story is so confusing that I may want to 
> deny the data.  How sure are you that your friend's story is true?  Were 
> you there?"
>
> Therefore did I devise this saying, to chide myself for having failed to 
> distinguish truth from falsehood:  "Your strength as a rationalist is your 
> ability to be more confused by fiction than by reality."

Okay, but you could have produced other maxims, or rules of thumb for
yourself from the same story, it seems to me.  This story, this personal
experience doesn't, to me anyway, make clear that level of confusion is
necessarily going to be a useful indicator to all rationalists as to how
rational they are being. Its neither evidence for or against that 
proposition
(to me anyway). It's just how you have characterised events for yourself.


>> In the recent discussion John C Wright finds
>> god, Damien didn't forget all the novels he had read, the movies he had
>> seen etc in couching his arguments to John C Wright, quite to the
>> contrary, he integrated his understanding of such cultural biases, and
>> pointed out that John C Wright had had the sort of experience that 'fit'
>> with his culture rather than one that would have 'fit' with a different
>> culture.
>
> The logical form of Damien's argument was that since Wright's purported 
> story, which might be real and might not be, was drastically inconsistent 
> with experiment, and drastically consistent with known fictions, it was 
> probably also a fiction.
>
> This doesn't mean we are reasoning from fictions as if they were real 
> events.  It means we are being aware of the probable causes of human 
> delusion.  But it is necessary to first investigate the question of 
> consistency with science; even true statements will often have some 
> *vague* resemblance to fiction, because there are so many fictions out 
> there.

I sort of agree with the bit "But it is necessary..." but I can't 
completely,
not as a generalisation for all rationalists. Rationalists don't start with
a mastery of all science as such would be understood if all the experiences
of all leading scientists in every field were available to them as each 
individual
rationalist as their own personal experiences. Individual rationalists can 
only
have individual scientific worldviews that are works in progress for them.

>> My explanation was only provisional so if it happens I'll be open to 
>> alternative explanations. And if it happens I won't have to throw away
>> all my experiences or forget stuff to explain it. I will only have to
>> change my model and I'll only have to change it in certain ways.
>
> If anyone ever wakes up with a blue tentacle, then you were virtuous to 
> claim in advance that the event was explicable.  If the real chain of 
> events leading up to the blue tentacle matches your given reason, then you 
> were virtuous to claim that reason as your specific explanation.
>
> If no one ever wakes up with a blue tentacle, then clearly a blue tentacle 
> wasn't the sort of thing which would ever be woven into reality by a 
> sequence of events that would constitute an "explanation" of it, and it 
> was a mistake to claim that a blue tentacle was an explicable event.

Or no one ever cared to go to the trouble of surprising someone else
by arranging for them to be awoken with a blue tentacle attached.

> What would be your explanation if one day, everyone in the world began 
> observing that two of something plus two of something made five of 
> something?


That's a good differentiating question.

That is not possible in *our* (which includes my) universe. One of the
obvious reasons it is not possible is that I am in the set of "everyone".
I can no more imagine a circumstance where two and two make five
than I can image a square with three-and-only-three sides.

Mathematical truths exist in a domain separate to the world of scientific
observation. (And separate to the vagaries of particular languages like
English, French, German etc). That 1 and 1 make 2  is fundamental.
2 and 2 making 4 is simply a very minor 'development' of the same
fundamental. People can, and do, (as you know), play games with
changing the words and symbols and putting different meanings on
symbols in different contexts but the underlying concepts of mathematics
are separate to the arbitrary selection of words we use to describe them.
ie. I can sort of imagine a world where people used the *word* five to
mean the concept four. Sort of like Shakespeare's Juliet could imagine
Romeo not be called Romeo and not being essentially different -
"what's in a name?". But I can't imagine a world in which *I* would
not know that the concept of two and two make four.

Nor, to go off on a slight tangent, can I imagine science or language
as inter-subjective communions between two or more agents that
don't get that one and one make two in any communion involving
myself.

>>> When you have only a poor explanation, one that doesn't make things 
>>> ordinary in retrospect, just admit you don't have an explanation, and
>>> keep going.  Poor explanations very, very rarely turn out to be
>>> actually correct.
>>
>> I don't think that this is right, or that it is a logical conclusion to
>> draw from the better parts of your argument in your essay. We have maps
>> of the terrain of reality because we need them. Maps have utility. If you 
>> give me a poor map and I know nothing of you and find that the map
>> is wrong then, in that case yes, perhaps I might be better off without
>> that map altogether, but if the map I have is one that I have constructed 
>> myself, then when I find it differs from the terrain I can just correct 
>> or improve the map.
>
> That is an argument for:  "I will sit down and write a story, knowing it 
> to be fiction, about how a secret organization came into my apartment and 
> replaced my arm with a blue tentacle.  I do not *believe* this has 
> happened.  No, seriously, I don't believe it and I'm not going to act as 
> if I believed it, because it's a stupid explanation.  But maybe the mental 
> exercise will shake something loose in my mind, and I'll think of a better 
> explanation."

I don't think my comments amount to that argument at all.

Perhaps you proposed a scenario that you thought was essentially as
obviously impossible as asking someone to imagine that 1 and 1 make 2,
but that in fact you didn't.

I think (but don't know for sure) that it *might* be possible in the future
to replace *your* arm with a blue tentacle while *you* sleep. This is not
an aim I imagine that I or anyone else aspires too. But on my (limited)
understanding of science and the real world currently I could not honestly
assert that that would be impossible so far as I know.

Perhaps I will learn something in future that will show it to be impossible
but at this stage I'm not at that point.

> To say that it can have utility to mentally extrapolate the consequences 
> of a premise is not the same as believing that premise.  One must be 
> careful here; if you act like you believe something, or if you end up 
> emotionally attached to the belief, I don't credit you as a rationalist 
> just because you claim you didn't believe you would win the lottery, you 
> bought the tickets "for the drama" of it, etc.  People with a fragmentary 
> understanding of the Way sometimes anticipate that they can pass as 
> rationalists by claiming not to believe the things they anticipate.
>
>>> A gang of people sneaking into your room with unknown technology is a
>>> poor explanation.  Whatever the real explanation was, it  wouldn't be
>>> that.
>>
>> I think you can only establish that it's poor (for others than you) in 
>> relation to the provision of a better one.  "I don't know", whilst a
>> fair and honest answer, is not any sort of explanation. My answer shows 
>> you I don't know but doesn't leave you (or importantly) me merely and
>> completely bewildered. It gives me things to check.
>
> That is not an *answer*.

It is an answer in the sense that if I perceive you asking a question in
a spirit of truthseeking and offer you something to work or play with
rather than dead silence, even though the something is not factually
correct or necessarily true the answer is an honest human response.
A social offering. I didn't lie to you or myself. Nor did I commit a lot
of time. I just responded honestly and socially rather than intellectually.
In so doing I gave you something to work with other than just your own
speculations.

It is one thing, a good thing, to understand Bayes theorem and to be able
to apply it (and teach it), it is another thing, also a good thing to try 
and
push outwards the boundaries of what is known or understandable or can
be made common between rational minds, and yet it is a third thing to
succeed in the second thing rather than to just attempt it.

I give you credit (in terms of goodwill) for trying to push back the
boundaries in important areas but that isn't the same as credit for
succeeding.

> It is not something to which Bayesian reasoning gives a high probability. 
> That is a science fiction story, a tool for brainstorming an answer.  I 
> have sometimes derived interesting ideas from my attempts to write SF, but 
> I know those stories for fiction, not reality.
>
> If you see something that looks like a poor explanation but is the only 
> explanation you have, it may take a bit of effort to achieve a state of 
> mind where you *really* don't anticipate it - rather than claiming to 
> yourself that you are dutifully skeptical.
>
>> And it is very hard for us as individuals to take other's
>> "rationalities" as givens when we don't get to see the others
>> observations as our own observations. Second (or more) hand
>> "observations" have to be discounted to some extend on first hand ones.
>
> See Robin Hanson and Tyler Cowen's paper on meta-rationality, "Are 
> Disagreements Honest?"  http://hanson.gmu.edu/deceive.pdf

I did. Its a good paper.

>> Progress depends on people (as change agents) being willing to stick
>> their necks out to try to explain.
>
> That doesn't require that you bet on, anticipate, or believe a hypothesis, 
> before it is confirmed.  It means that you write science fiction about a 
> poor hypothesis, to get your mind working on the problem.

I was going to agree with this last, however, sometimes it does make sense
to bet on a hypothesis that is merely a hypothesis.

Brett Paatsch