[extropy-chat] No rejection of science! Re: SI morality

Eliezer S. Yudkowsky sentience at pobox.com
Fri Apr 30 16:05:29 UTC 2004

Hal Finney wrote:
> I think my entire post can go through unchanged if you simply substitute
> "the practice of science" or "the scientific consensus" for the shorthand
> "science" everywhere.

These are extraordinarily different things.

The practice of science is a social process.
The consensus of science is an opinion poll.

The actual working part of science is Bayesian probability theory, which 
individual scientists and their social dynamics partially and imperfectly 

> It is this approach, of taking scientific things one at a time, which
> I think is dangerous.  You are setting your judgement above that
> of the scientific community.  It is dangerous to try to cherry-pick
> among scientific issues to decide for which ones we will agree with
> the consensus and for which ones we will decide that the scientific
> consensus is wrong.  This makes it too easy for our individual lapses
> in rationality to find an excuse to enter and influence our thinking.

Science intrinsically requires individual researchers setting their 
judgment above that of the scientific community.  The social process of 
science encourages people to do the work and recognizes when they have 
done the work.  The social process is not an actual human brain, has not 
the power of intelligence.  If individuals do not have novel opinions and, 
yes, disagreements, for the scientific process to recognize as correct, 
there is no science.

We might not know that Einstein was right about relativity if there were 
no scientific process to recognize him.  But Einstein would still be the 
one who thought through to the answer.  Yes, Einstein relied on a social 
process of science to recognize Maxwell, but it is still humans who power 
the engine.

The overall rationality of academia is simply not good enough to handle 
some necessary problems, as the case of Drexler illustrates.  Individual 
humans routinely do better than the academic consensus.

> Essentially I am advocating the idea of following the scientific consensus
> faithfully; you might even say, blindly.  The reason is because our
> errors of rationality are so pervasive and seductive that we are more
> likely to be wrong than is the scientific consensus.
> This is not an easy principle to follow!  It feels like an abdication of
> responsibility, like an abandonment of critical thinking.  But when I look
> within, these feelings do not come from the part of me that loves truth,
> they come from the part of me that loves myself.  They are a manifestation
> of ego.  They come from an emotional desire to be the master of my fate,
> which means making my own decisions about what to believe and what not
> to believe.  Delegating these matters to any outside social institution,
> even one whose track record in approaching the truth is greater than
> anything mankind has ever developed, goes against powerful mental
> instincts.  Nevertheless I claim that this is what we have to try to do.

Yes, the Way of rationality is difficult to follow.  As illustrated by the 
difficulty that academia encounters in following.  The social process of 
science has too many known flaws for me to accept it as my upper bound.

Academia is simply not that impressive, and is routinely beaten by 
individual scientists who learn to examine the evidence supporting the 
consensus, apply simple filters to distinguish conclusive experimental 
support from herd behavior.  Robyn Dawes is among the scientists who have 
helped document the pervasiveness of plausible-sounding consensuses that 
directly contradict the available experimental evidence.  Richard Feynman 
correctly dismissed psychoanalysis, despite the consensus, because he 
looked and lo, there was no supporting evidence whatsoever.  Feynman tells 
of how embarassing lessons taught him to do this on individual issues of 
physics as well, look up the original experiments and make sure the 
consensus was well-supported.

Given the lessons of history, you should sit up and pay attention if Chris 
Phoenix says that distinguished but elderly scientists are making blanket 
pronunciations of impossibility *without doing any math*, and without 
paying any attention to the math, in a case where math has been done.  If 
you advocate a blanket acceptance of consensus so blind that I cannot even 
apply this simple filter - I'm sorry, I just can't see it.  It seems I 
must accept the sky is green, if Richard Smalley says so.

I can do better than that, and so can you.

Eliezer S. Yudkowsky                          http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence

More information about the extropy-chat mailing list