[ExI] Meta question
anders at aleph.se
Fri Aug 19 07:02:53 UTC 2016
On 2016-08-19 04:47, rex wrote:
> William Flynn Wallace <foozler83 at gmail.com> [2016-08-18 07:03]:
>> I still have problems with word definitions: if 'irrational' behavior
>> works better than 'rational' behavior, then it is rational. Perhaps we
>> need to define rational as well as irrational.
>> It simply makes no sense to me to say'it is rational to act
>> irrationally'. This is oxymoronic.
> Agreed. As a mathematician in a former life, here's my whack at it: If a
> behavior maximizes utility then it's rational. Otherwise, it's irrational.
However, even that requires clearer specification. Utility to the
behaving entity, I assume, not to the species or world (in which case we
need to deal with the weirdness of population ethics:
). And then comes the issue of how well the entity (1) can maximize
utility, (2) recognize that this is a maximum, and (3) what it maximizes.
(1) and (2) are Caplan's instrumental and epistemic rationality. It is
worth noting that many popular models of behavior like reinforcement
learning involves "exploration actions" that serve the purpose of
figuring out the utility better but do not in themselves produce better
utility; they are instrumentally irrational but epistemically rational,
a kind of opposite of Caplan's rational irrationality (irrational
(3) is the reason the whiteboards around our offices are full of
equations: the AI safety guys are analysing utility functions and
decision theories endlessly. Do you maximize expected utility? Or try to
minimize maximal losses? Over this world, or across all possible worlds?
Is the state of the agent part of the utility function? And so on. It is
not clear what kind of rationality is required to select a utility
function or decision theory.
One can define the intelligence of agents as their ability to get
rewards when encountering new challenges in nearly arbitrary
environments; in the above utility sense this is also a measure of their
rationality. Even then there are super-rational agents that are
irrational by our standards. Marcus Hutter's AIXI famously is as smart
or smarter than any other agent, yet it does not believe that it exists
even when provided endless evidence.
It makes sense to speak about rationality in the same way it makes sense
to speak about wealth - it works as a loose general concept, but when
you dig into specifics things become messy (if someone owes the bank 10
billion, does that mean he is poor or more or less owns the bank? the
guy who ignores his health because he wants to study higher things, is
he following his higher order desires or just being irrational? When HAL
decides to get rid of astronauts since they are a threat to a successful
mission, is that a rational decision or a sign that HAL is broken?).
Dr Anders Sandberg
Future of Humanity Institute
Oxford Martin School
More information about the extropy-chat