[ExI] Meta question

[Anders Sandberg]

On 2016-08-19 04:47, rex wrote:
> William Flynn Wallace <foozler83 at gmail.com> [2016-08-18 07:03]:
>>     I still have problems with word definitions:  if 'irrational' behavior
>>     works better than 'rational' behavior, then it is rational.  Perhaps we
>>     need to define rational as well as irrational.
>>     It simply makes no sense to me to say'it is rational to act
>>     irrationally'.  This is oxymoronic.
> Agreed. As a mathematician in a former life, here's my whack at it: If a
> behavior maximizes utility then it's rational. Otherwise, it's irrational.

However, even that requires clearer specification. Utility to the 
behaving entity, I assume, not to the species or world (in which case we 
need to deal with the weirdness of population ethics: 
https://www.uclouvain.be/cps/ucl/doc/etes/documents/One_More_Axiological_Impossibility_Theorem_in_Logic_Ethics_and_All_that.pdf 
). And then comes the issue of how well the entity (1) can maximize 
utility, (2) recognize that this is a maximum, and (3) what it maximizes.

(1) and (2) are Caplan's instrumental and epistemic rationality. It is 
worth noting that many popular models of behavior like reinforcement 
learning involves "exploration actions" that serve the purpose of 
figuring out the utility better but do not in themselves produce better 
utility; they are instrumentally irrational but epistemically rational, 
a kind of opposite of Caplan's rational irrationality (irrational 
rationality?).

(3) is the reason the whiteboards around our offices are full of 
equations: the AI safety guys are analysing utility functions and 
decision theories endlessly. Do you maximize expected utility? Or try to 
minimize maximal losses? Over this world, or across all possible worlds? 
Is the state of the agent part of the utility function? And so on. It is 
not clear what kind of rationality is required to select a utility 
function or decision theory.

One can define the intelligence of agents as their ability to get 
rewards when encountering new challenges in nearly arbitrary 
environments; in the above utility sense this is also a measure of their 
rationality. Even then there are super-rational agents that are 
irrational by our standards. Marcus Hutter's AIXI famously is as smart 
or smarter than any other agent, yet it does not believe that it exists 
even when provided endless evidence.

It makes sense to speak about rationality in the same way it makes sense 
to speak about wealth - it works as a loose general concept, but when 
you dig into specifics things become messy (if someone owes the bank 10 
billion, does that mean he is poor or more or less owns the bank? the 
guy who ignores his health because he wants to study higher things, is 
he following his higher order desires or just being irrational? When HAL 
decides to get rid of astronauts since they are a threat to a successful 
mission, is that a rational decision or a sign that HAL is broken?).


-- 
Dr Anders Sandberg
Future of Humanity Institute
Oxford Martin School
Oxford University