[ExI] AI alignment

[Keith Henson]

I am not a figure in the AI safety world (now called alignment), but I
have watched things develop for over 40 years.

The people who worked on the subject are frighteningly smart.  And
they have not made notable progress.  My suspicion is that it may not
be possible for humans to solve the alignment problem, in fact, it may
be something that cannot be solved at all.

Part of the reason is that a very well-aligned (but useful) AGI
combined with human desires could have undesired effects. About 18
years ago on the sl4 list, I wrote an AI story.  “The Clinic Seed” was
about a well-aligned (friendly) nanotech-based medical AGI which (due
to human desires to live an easy life) made a population go
biologically extinct (they all uploaded).

I suspect that a useful AGI combined with human desires potentially
leads to this kind of extinction.  I also suspect that an AGI with
limitations to keep it from responding to human requests of this kind
would be catatonic or perhaps useless.

There were no follow-ups to the story.  Why I am not entirely sure,
but it could be that the sl4 people did not want to think about a
seemingly safe AI that, due to *human* characteristics, could result
in such an outcome.

There are negative aspects of waiting for people to solve the
alignment problem.  Presumably, AGI will help solve all kinds of
medical problems, even leading to effective immortality.  Delaying the
development condemns millions, perhaps billions to death.

Perhaps instead of worrying about the human species surviving, we
should see what we might be able to do in shaping our successors.



Keith