[ExI] Eliezer Yudkowsky on FB about why we're all doomed

[Dave Sill]

Back in the golden days of this list, Eliezer was a regular contributor.

*The global economy finally got shut down for a year by a human-boosted
pathogen and it wasn't done by a military faction, an evil corporation bent
on profit, one suicidal researcher, or done deliberately at all.  We'll
probably never know for certain, but it's now looking 80% probable to me
(and eg Nate Silver says he's at 60%):  Covid-19 was accidentally released
from a US-funded Chinese laboratory doing gain-of-function studies.
Presumably, this accident was carried out by researchers looking for a good
paper they could get into a journal; as incentivized by journal editors
trying to play the role of scientists, and not otherwise incentivized to
guard the earth from harms that they didn't imagine would be traced back to
them personally.*

*And the journal editors were right: even after it has become known as the
dominant probability, to those who can know probabilities, that Covid-19
was a human-produced catastrophe, the journal editors and funding agencies
and promotion panels (who are the closest thing to a locus of
responsibility short of voters) will get off scott-free just the way their
hindbrains anticipated.  Except for those journal editors having to spend a
year in lockdown along with the rest of us, of course.  But that's not a
positional loss, a status loss, so I doubt it registers much with them as a
regret even in retrospect.  Unless any close friends of theirs died of
Covid, and in that case I expect they managed to avoid mental pain through
denying all responsibility - it wasn't them who spilled the beaker, after
all!  In fact, I'd guess that the journal editors who published
gain-of-function studies have yet to think that it might be their locus of
responsibility at all - how is it *their* fault for incentivizing a
wide-enough group of applicants that the research would inevitably be
carried out?*

*Could this disaster have been avoided?  Not in this Earth or any of its
close counterfactual neighbors.  You can imagine petty hindsight-based
time-travel interventions like stepping into the Wuhan Institute of
Virology at the right moment and preventing one spill, one animal bite.
But to prevent the disaster without a time machine or benefit of hindsight,
you'd have to shut down the journal editors providing a distributed
community of scientists with prestige incentives for doing gain-of-function
research.  Prohibiting this type of research inside the USA simply resulted
in US funding agencies moving the work to China under lower biosafety
conditions, which is itself an exemplar of how people doing symbolic stands
against problems are unable to solve them (and don't really care).*

*This is not something that effective altruists could have prevented even
with a billion dollars in funding, as I read it.  The journal editors
incentivizing distributed attempts at gain-of-function research were not
set up in a way where effective altruists could stop them from doing what
they do.  Earth is just not set up to make it easy or even possible to
channel "concern over human-made pathogens" into anything effective, rather
than various symbolic regulatory stances and ethics pronouncements that
didn't prevent the accident even though they looked like they were "doing
something".  And of course the collective vague notion and practice of
"bioethics" was an active hindrance to vaccine testing, economical vaccine
scaling, and rapid Covid testing early in the pandemic.*

*The fundamental flaw, I would say, is that even the people "doing things"
about pandemics were not running an algorithm where they stopped and
checked "But will this actually effectively prevent <a scenario like
Covid-19>?" and instead just did things and called that their proud day's
work.  It's very uncomfortable and unnatural for people to think like that,
since even after you've prohibited a form of the research inside the USA,
somebody just comes back and says "Okay it's going to move to China under
lower biosafety conditions, the problem is not solved and your moral stance
didn't even help with it" and you don't get to feel good about all that
hard work you did.*

*And almost nobody thinks like that, unless their personal status or
existing wealth is on the line in a very direct way.  Before the planetary
lockdown actually happens, it hardly seems like a thing that actually
happens to you in the real world, as opposed to the words "biotech
disaster" being a political football for political games.  A biotech
accident putting the world on lockdown for a year?  Ha ha, lol, says the
brain; it just gets processed as a fanciful tale (or morality-tale if
you're more sympathetic).  It doesn't seem real until it goes and actually
happens.  I suspect that it doesn't seem real even after it happens.*

*The Covid-19 pandemic was caused by an accidental release by a US-funded
Chinese lab doing gain-of-function studies on human transmission, and there
is no way in this Earth that this Earth could have decided to instead not
do that, even if every effective altruist in the world had focused on
biosafety alone.*

*Even with the global economy having been partially shuttered for a year
and best-estimates of 10 million excess dead, I will be pleasantly
surprised (though not shocked) if gain-of-function research gets shut down
even *now*, well after it's become clear that the costs far exceeded the
actual gains.  All the gain-of-function research on coronaviruses helped
little or not at all with Covid-19, so far as I can tell.  And of course
nobody will learn any lessons at all that generalize beyond biology.  I
mostly don't expect them to learn the lesson about biology, narrow as that
one may be.  Maybe we'll see gain-of-function studies stopped for
coronaviruses in particular, though this is more generated by the thought
"What would be the least dignified response we could see" rather than a
prediction per se.*

*Now, you may wonder: why am I bothering to post about an issue this
trivial?*

*And of course the answer is: because of the way in which it stands as an
enormous blatant case-in-point for AGI ruin.  All this is not *exactly* how
I expect Artificial General Intelligence to wipe out all life on Earth, one
day (or rather one minute) in the future after an unknowable number of
preceding days.  But it is a useful exemplar of how these things play out
in life as opposed to morality.  The world will end, in the end, because
some researchers and philanthropists and journal editors have local
prestige incentives for AGI gain-of-capability studies; and don't imagine,
while most of the event is playing out, up until the very end and maybe not
even then, that they personally will wipe out all life on Earth, or be held
accountable for its extermination.  The notion of all life being wiped out
won't sound like a positional status loss, so won't really register with
them.  Maybe if they have kids - but probably not even then, because all
mental pain can be avoided by telling themselves they won't do it or won't
be at fault.  It won't be processed inside the genre of reality before it
actually happens, and that part will be brief.*

*If you have any fantasies about better collective decision-making than
that, compare them to how people in real life actually acted about human
challenge trials of vaccines (namely not doing them despite the idea being
very clear and obvious and spoken up early) or doing earlier scaling of
more vaccines for mass production.  Oh, to be sure, some individuals in the
AGI ruin industry are a little smarter than that, and have a little more
concentrated power; but also to be sure, averting AGI ruin is far far
technically harder.  In AGI alignment there is not actually any equivalent
of "Deploy rapid tests immediately without medical gatekeepers, run human
challenge trials of all the vaccines, immediately invest massive amounts
into scaling vaccine production at the $500/dose price level instead of the
$10/dose price level because shutting down the economy is so much more
expensive than that".  There is not a thing we know how to do that would
solve the problem, which just needs to actually be done.  So expect
symbolic acts of ethics that are not actually effective, that do not pass
the test of "but does this actually prevent the ruination"; including from
the relatively smarter empowered individuals in the field, and from the
effective altruists hovering nervously around the oncoming death.*

*As for the notion that AGI ruin will be enacted by the military, by a
corporation bent on dirty profit, by evil terrorists, by $hatedoutgroup, by
a deliberately suicidal group - those are fairytales.  It is a thought born
of wanting there to be a moral lesson in dire prophecies rather than a
neutral causal extrapolation of what the future will look like if it
resembles past stories.  Contrast the fairytales about biotech global
catastrophic risks to the actual biotech global catastrophe, which happened
not in the most dramatic way it could happen, but in the easiest and in
some sense saddest and least dignified way it could happen.  The part where
it's "a US-funded Chinese laboratory" is oddly symbolic of how much real
history ends up not sounding like the story that any particular ideological
storyteller would invent.  Notice the difference between the morality-tales
of futurism meant to elevate or lower the status of particular groups, or
impel people to symbolic acts of fake prevention; versus the sad little
farces of unsteerable reality, where the world economy gets shut down by an
accident, and the perpetrators didn't even want that or buy Zoom stock as
they did it.*

*The social, organizational, political story of the future will look much
like the past, I expect, up until the last end.  Here then is the past, if
you care to generalize from it at all.*


*(#long #essay #AGI #AGIruin #Covid19)*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20210526/d2be635a/attachment.htm>