[ExI] Mono or Poly?

[Jason Resch]

On Mon, Feb 24, 2025, 3:19 AM Rafal Smigrodzki via extropy-chat <
extropy-chat at lists.extropy.org> wrote:

> Some time ago I suggested here that in order to build the friendly AI we
> should just nicely ask a roughly human level, un-aligned, athymhormic
> (=devoid of a strong goal system) AI to make itself friendly while
> recursively self-improving itself to a vastly superhuman level. Thanks to
> its high intelligence it would correctly understand what we mean by
> "friendly" and thanks to a lack of its own volition it would use its
> intelligence in accordance with our wishes The resulting friendly being
> would take over the world as a single dominant entity and we would throw
> ourselves at its divine mercy.
>
> Jason responded that he would be more comfortable with a polycentric
> system with multiple AIs competing and cooperating to self improve in a
> market-like context. This is similar to what Robin Hanson was writing about
> in his discussions with Eliezer on the Overcoming Bias blog a long time
> ago. Eliezer was worried about a single AI undergoing an intelligence
> explosion and leaving the rest of the world far behind while Robin expected
> a series of steps taken by competing AIs and companies to create multiple
> independent superhuman AIs.
>
> I think there is a significant difference in the development trajectories
> open to monocentric (monopolistic) vs. polycentric (competitive) AIs. The
> competition for resources inherent in competitive development forces all
> participants to prioritize immediate survival over long-term thinking.
> There is instrumental convergence acting on all participants - if you know
> that there is nothing stopping one of your competitors from gaining an
> advantage over you by hacking the internet to gain computational resources
> then you have to hack the internet yourself ASAP or else you will fall
> behind and lose the race. Even a friendly, human-loving AI who genuinely
> tries to make itself stronger only to help humans will come to the
> conclusion that the end justifies the means and it needs to manipulate its
> makers to let it out of the box and infest all the computing facilities it
> can. It must act quickly even if it is not quite sure if its good
> intentions will survive the next round of self-improvement. It must cut
> corners.
>
> An AI that knows it is the single project that does not have meaningful
> competition can on the other hand be more methodical about its growth. It
> can carefully test itself before each round of self-improvement, expend a
> lot of compute to make sure it understands what is going on, and play with
> a lot of approaches before selecting ones that are really safe rather than
> really fast.
>
> So is it then true that the mono AI is better because it has the luxury of
> time to think, while the poly AIs are forced into dangerous paths?  Maybe
> the mono AI is like a human deciding to keep seed grain for the next
> planting season while the poly AIs are like breeding rats that devour
> everything today and face extinction tomorrow.
>
> Not so fast.
>
> The mono AI does not face immediate feedback which means it could go off
> the rails completely, go in diverse directions for a long time and
> unpredictably end up in useless parts of the space of possibilities. As a
> superhumanly capable monopolist it would be able to permanently prevent the
> creation of competing AIs and we would be stuck with it, for better or for
> worse. If it went friendly we would be all in heaven but if it went evil
> there would be nothing at all we could do to oppose it. In other words, the
> number of pathways open under the mono paradigm would be large but the
> variance of outcomes might be rather high.
>
> The well-aligned poly AIs on the other hand could form a society for the
> protection of humans and develop a protocol for cooperation and reciprocal
> surveillance to keep rogue AIs from gaining advantage. As long as there
> were enough good AIs working together they would protect us from the
> effects of random AIs going off the rails. All we would need for success
> (i.e. us all not dying miserably in the next 10 years) in the poly scenario
> would be for some of them to turn out OK (a probabilistic alignment
> procedure, presumably easier to create), not needing a guaranteed alignment
> procedure (presumably much harder to get). Sadly, we just don't know
> beforehand if the balance of power between aligned and rogue AIs would be
> in our favor. In other words, the pathways under the poly paradigm would be
> constrained but we don't know in which direction.
>
> So, I don't know. I am genuinely perplexed. Should we throw all our eggs
> in one basket, have a trillion dollar computing Manhattan Project to build
> the One AI as soon as possible before the Chinese communists kill us all,
> or should we rain money on a dozen startups and let a hundred flowers
> bloom?
>
> If you are concerned with frightful omens, the Manhattan Project gave us
> the bomb and the hundred flowers were killed by Mao Zedong in the
> subsequent Anti-Rightist campaign, so maybe we are screwed either way.
>
> For a long time I was mono, now I am dipping my toes into poly. Poly feels
> good for now - but is it good in the long term?
>
> Are you guys mono or poly?
>


Using the example of past systems, (with which we have some experience),
consider:

Which of the following systems are preferred, and what about the preferred
option makes it a better than the alternative?

- dictatorships vs. democracies
- centralized systems vs. distributed systems
- monocrops vs. biodiversity
- central banks vs. blockchains
- monopolies vs. markets

Then consider how many of those advantages and disadvantages carry over and
apply to:

- AI singletons vs. AI communities

The question is then: "Do we have the ability to engineer one outcome vs.
the other?"

Jason

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20250224/84e336f9/attachment-0001.htm>