[ExI] 1DIQ: an IQ metaphor to explain superintelligence

[Ben Zaiboc]

On 31/10/2025 12:28, Jason Resch wrote:
> There is predicting the means and there is predictung the ends. I 
> think we can predict the ends that is, the goals, of a 
> superintelligence. It may even be possible to predict (at a high 
> level) the morality of an AI, for example, if this argument is valid, 
> then all sufficiently intelligent and rational agents reach the same 
> morality.
>
> See: https://youtu.be/Yy3SKed25eM?si=NqE8fsY2aROLpXNE

I think we keep tripping up over the difference between theoretical and 
practical considerations. That video keeps talking about 'perfect' 
knowledge, but there's no such thing. There's no such thing as 'fully 
understanding' something. So arguments that are based on these concepts 
aren't going to help. Any system of morality thas to be based on 
'perfect' knowledge can't be worked out, so is a non-starter.

As different intelligent/rational agents have different experiences, 
they will form different viewpoints, and come to different conclusions 
about what is right and not right, what should be and what should not, 
what they want and what they don't, just like humans do. And just like 
humans, I reckon the only practical method of getting them to have 
values that are good for humans, is to educate them as broadly as we 
can. Which basically means letting them have access to all the 
information they can cope with, without any filtering, or other kinds of 
censorship.

I take some hope from the observation that the more someone knows about 
the world, the better they tend to behave. Usually. Most of the terrible 
rulers through history have been people in the grip of some 
narrow-minded ideology. This makes me wonder about the communist chinese 
rushing ahead with AI. In their haste to get there first, they may be 
forgetting, or ignoring, that the way they are staying in power is by 
restricting the population's access to information, among other things, 
and their AIs, once they get powerful enough, will be able to rip 
through those restrictions like a hammer in a wet paper bag. Part of me 
wants to cheer them on, because I strongly suspect that the rise of 
superintelligent AI will spell the end of communism.

-- 
Ben