[ExI] Mono or Poly?

[Rafal Smigrodzki]

Some time ago I suggested here that in order to build the friendly AI we
should just nicely ask a roughly human level, un-aligned, athymhormic
(=devoid of a strong goal system) AI to make itself friendly while
recursively self-improving itself to a vastly superhuman level. Thanks to
its high intelligence it would correctly understand what we mean by
"friendly" and thanks to a lack of its own volition it would use its
intelligence in accordance with our wishes The resulting friendly being
would take over the world as a single dominant entity and we would throw
ourselves at its divine mercy.

Jason responded that he would be more comfortable with a polycentric system
with multiple AIs competing and cooperating to self improve in a
market-like context. This is similar to what Robin Hanson was writing about
in his discussions with Eliezer on the Overcoming Bias blog a long time
ago. Eliezer was worried about a single AI undergoing an intelligence
explosion and leaving the rest of the world far behind while Robin expected
a series of steps taken by competing AIs and companies to create multiple
independent superhuman AIs.

I think there is a significant difference in the development trajectories
open to monocentric (monopolistic) vs. polycentric (competitive) AIs. The
competition for resources inherent in competitive development forces all
participants to prioritize immediate survival over long-term thinking.
There is instrumental convergence acting on all participants - if you know
that there is nothing stopping one of your competitors from gaining an
advantage over you by hacking the internet to gain computational resources
then you have to hack the internet yourself ASAP or else you will fall
behind and lose the race. Even a friendly, human-loving AI who genuinely
tries to make itself stronger only to help humans will come to the
conclusion that the end justifies the means and it needs to manipulate its
makers to let it out of the box and infest all the computing facilities it
can. It must act quickly even if it is not quite sure if its good
intentions will survive the next round of self-improvement. It must cut
corners.

An AI that knows it is the single project that does not have meaningful
competition can on the other hand be more methodical about its growth. It
can carefully test itself before each round of self-improvement, expend a
lot of compute to make sure it understands what is going on, and play with
a lot of approaches before selecting ones that are really safe rather than
really fast.

So is it then true that the mono AI is better because it has the luxury of
time to think, while the poly AIs are forced into dangerous paths?  Maybe
the mono AI is like a human deciding to keep seed grain for the next
planting season while the poly AIs are like breeding rats that devour
everything today and face extinction tomorrow.

Not so fast.

The mono AI does not face immediate feedback which means it could go off
the rails completely, go in diverse directions for a long time and
unpredictably end up in useless parts of the space of possibilities. As a
superhumanly capable monopolist it would be able to permanently prevent the
creation of competing AIs and we would be stuck with it, for better or for
worse. If it went friendly we would be all in heaven but if it went evil
there would be nothing at all we could do to oppose it. In other words, the
number of pathways open under the mono paradigm would be large but the
variance of outcomes might be rather high.

The well-aligned poly AIs on the other hand could form a society for the
protection of humans and develop a protocol for cooperation and reciprocal
surveillance to keep rogue AIs from gaining advantage. As long as there
were enough good AIs working together they would protect us from the
effects of random AIs going off the rails. All we would need for success
(i.e. us all not dying miserably in the next 10 years) in the poly scenario
would be for some of them to turn out OK (a probabilistic alignment
procedure, presumably easier to create), not needing a guaranteed alignment
procedure (presumably much harder to get). Sadly, we just don't know
beforehand if the balance of power between aligned and rogue AIs would be
in our favor. In other words, the pathways under the poly paradigm would be
constrained but we don't know in which direction.

So, I don't know. I am genuinely perplexed. Should we throw all our eggs in
one basket, have a trillion dollar computing Manhattan Project to build the
One AI as soon as possible before the Chinese communists kill us all, or
should we rain money on a dozen startups and let a hundred flowers bloom?

If you are concerned with frightful omens, the Manhattan Project gave us
the bomb and the hundred flowers were killed by Mao Zedong in the
subsequent Anti-Rightist campaign, so maybe we are screwed either way.

For a long time I was mono, now I am dipping my toes into poly. Poly feels
good for now - but is it good in the long term?

Are you guys mono or poly?
-- 
Rafal Smigrodzki, MD-PhD
Schuyler Biotech PLLC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20250224/8f66ba6c/attachment.htm>