[ExI] AGI will likely be uncontrollable and very strange

[John Clark]

On Sat, Oct 24, 2026 at 9:53 AM Jason Resch via extropy-chat <
extropy-chat at lists.extropy.org> wrote:\\\


*> Should these ethics be Utilitarian in nature or more like Kantian
> categorical imperatives?*


*Short answer, Utilitarian.  *

*Longer answer: Kant thought you should act only according to a rule that
you could rationally want everyone to follow rather than on the
consequences of an individual action. And I think that's nuts. If being
moral doesn't result in a minimization of suffering and a maximization of
intelligence (and therefore of its inevitable byproduct consciousness) then
morality has no utility, and if it has no utility then there is no point in
me being moral. *

*Kant believed that, although he couldn't list them, there existed a finite
set of intuitive ethical axioms that one should ALWAYS follow without
exception, that were logically self consistent and covered every possible
situation. He believed that "don't lie" was such a categorical imperative
but if an SS officer asked "do you know where Anne Frank is?" the
intuitively obvious moral thing to say is NOT "yes, she's right upstairs".
And the trolley problem vividly illustrates the intuitively obvious moral
thing to do is not always the thing that leads to a minimization of
suffering or a maximization of consciousness. *

*This moral ambiguity really shouldn't have been a surprise, thanks to
Gödel we know that it's impossible to even make arithmetic both logically
complete and consistent, it's unrealistic to expect we could do better with
ethics.  *

*> Smullyan comes up with a wonderful definition of the Devil: the
> unfortunate length of time it takes for sentient beings as a whole to come
> to be enlightened.*


*I think it's possible to become a little more specific than that. The
Devil is the length of time it takes for a sentient being to realize
Darwinian Evolution exists and is incredibly cruel. *

* > Truth and beauty are different sides of the same*


*It's true that if you're beautiful to the opposite sex then the chances of
your genes entering the next generation are improved, although if you're
spectacularly beautiful then you are also spectacularly conspicuous and
your chances of being eaten by a predator are also increased. *

*>If Jesus were real*


*God is real but Jesus is not because Jesus starts with a "J", so Jesus is
an integer unless declared to be real. It's been a long time but I still
remember a little Fortran.*


*John K Clark*

>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20261025/cc4f0574/attachment.htm>