[ExI] Zuboff's morality (Was: Re: 1DIQ: an IQ metaphor to explain superintelligence)

[Ben Zaiboc]

Ok, I've had a look at his paper, and made a few substitutions to make 
it easier to understand. Let me know if you object to any of these:

'desire' = intention
'belief' = anticipated result
'correctable' = changeable
'real' = preferred
'perfect grasp' = foreknowledge


The relevant passages now read, with my comments in brackets:

"Imagine that I have before me on a table a cup containing a thick, 
brown, steaming liquid.

I want to drink that stuff because I think it is hot chocolate. But it 
is actually hot mud. Well, in that case I don’t really intend to drink 
it. And neither is it in my self-interest to do so.

This example brings out the way in which intentions depend on 
anticipated results. I only ever intend to do a thing because of what I 
anticipate the result to be."

(this is not true. It's not uncommon to have an intention to do 
something in order to /find out/ what the result will be rather than in 
anticipation of an expected result. It would be more accurate to say 
that intentions CAN be based on anticipated results, and that you MAY do 
a thing because of the anticipated result. In Zuboff's original 
language, you would say 'to have a desire to form a belief about 
something'. The 'desire' precedes the 'belief', rather than the other 
way around, in this case. When A can cause B or B can cause A, you can't 
draw the conclusion that 'A depends on B')

"And since anticipated results are changeable, so are intentions."

(this implies that intentions are changeable /because/ anticipated 
results can change. It's possible to change your mind about the 
anticipated results of an intended action, or to change your intended 
action and anticipate the same result. It would be more accurate to say 
that both anticipations and intentions are changeable, but a change in 
one doesn't necessarily enforce a change in the other)

"From this observation I arrive at a sweeping principle: My only 
preferred intentions are those I would have if I had a foreknowledge of 
everything involved."

(because of the above, this is a false conclusion)

"If there is any intention I have only because my foreknowledge of the 
outcome is imperfect, then that cannot be among my preferred intentions."

(this would rule out any intention to find something out (because if you 
want to find something out, you necessarily don't already know the 
answer). I don't know about anyone else, but a lot of my 'preferred 
intentions' have the aim of finding things out that I don't already 
know. If you already knew, there would be no need to have an intention 
to find it out)

"And gratifying that intention cannot be in my preferred self-interest. 
The principle going along with this that governs my actions must tell me 
to act, as far as possible, as I would want myself to be acting with a 
foreknowledge of everything involved."

(it should be obvious now why this is nonsense, but nevertheless, let's 
follow this line of thought through (italics are mine):)

"This foreknowledge that defines my preferred intentions and my best 
course of action," /is of course impossible. He goes on to explain why/. 
"It would have to embrace not only the full experience, from behind the 
eyes (or other sensors), of every sentient being but also every 
potential development of experience. It would include within it, all the 
motivations of all of the various systems of intention" /which would 
simply conflict with each other. The overall result would be chaos and 
paralysis (in case this is not obvious, consider combining the 
motivations of a religious fundamentalist with those of a 
scientifically-literate materialist. These are conflicting value 
systems. Objective facts can't reconcile them. 'Perfect foreknowledge' 
can't do a thing when subjective values are involved. Let's say that you 
have the opportunity to punish/forgive someone who has stolen something 
from you. The values of one person (that you have, according to this 
theory, magical access to) dictate that the thief should be punished 
regardless of the circumstances of the crime, because 'STEALING IS 
WRONG'. You also have access to the values that tell you that stealing 
is often wrong, but can be forgiven under certain circumstances. How can 
there be any reconciliation of these two views? What facts can help?)/.

So even if there was any possibility of this, it still couldn't lead to 
any rational definition of morality. The requirement to know all 
possible points of view, and all outcomes of all actions are impossible 
enough, but add on top the requirement to /reconcile/ all points of 
view? And only then can you figure out what's good and what's bad?

Bonkers, pure and simple.

I trust you'll understand that this is as much as I was willing to read 
of the paper.

-- 
Ben