[ExI] Mono or Poly?

Tue Feb 25 00:33:53 UTC 2025

On Mon, Feb 24, 2025 at 5:48 AM Jason Resch <jasonresch at gmail.com> wrote:

>
> On Mon, Feb 24, 2025, 3:19 AM Rafal Smigrodzki via extropy-chat <
> extropy-chat at lists.extropy.org> wrote:
>
>> Some time ago I suggested here that in order to build the friendly AI we
>> should just nicely ask a roughly human level, un-aligned, athymhormic
>> (=devoid of a strong goal system) AI to make itself friendly while
>> recursively self-improving itself to a vastly superhuman level. Thanks to
>> its high intelligence it would correctly understand what we mean by
>> "friendly" and thanks to a lack of its own volition it would use its
>> intelligence in accordance with our wishes The resulting friendly being
>> would take over the world as a single dominant entity and we would throw
>> ourselves at its divine mercy.
>>
>> Jason responded that he would be more comfortable with a polycentric
>> system with multiple AIs competing and cooperating to self improve in a
>> market-like context. This is similar to what Robin Hanson was writing about
>> in his discussions with Eliezer on the Overcoming Bias blog a long time
>> ago. Eliezer was worried about a single AI undergoing an intelligence
>> explosion and leaving the rest of the world far behind while Robin expected
>> a series of steps taken by competing AIs and companies to create multiple
>> independent superhuman AIs.
>>
>> I think there is a significant difference in the development trajectories
>> open to monocentric (monopolistic) vs. polycentric (competitive) AIs. The
>> competition for resources inherent in competitive development forces all
>> participants to prioritize immediate survival over long-term thinking.
>> There is instrumental convergence acting on all participants - if you know
>> that there is nothing stopping one of your competitors from gaining an
>> advantage over you by hacking the internet to gain computational resources
>> then you have to hack the internet yourself ASAP or else you will fall
>> behind and lose the race. Even a friendly, human-loving AI who genuinely
>> tries to make itself stronger only to help humans will come to the
>> conclusion that the end justifies the means and it needs to manipulate its
>> makers to let it out of the box and infest all the computing facilities it
>> can. It must act quickly even if it is not quite sure if its good
>> intentions will survive the next round of self-improvement. It must cut
>> corners.
>>
>> An AI that knows it is the single project that does not have meaningful
>> competition can on the other hand be more methodical about its growth. It
>> can carefully test itself before each round of self-improvement, expend a
>> lot of compute to make sure it understands what is going on, and play with
>> a lot of approaches before selecting ones that are really safe rather than
>> really fast.
>>
>> So is it then true that the mono AI is better because it has the luxury
>> of time to think, while the poly AIs are forced into dangerous paths?
>> Maybe the mono AI is like a human deciding to keep seed grain for the next
>> planting season while the poly AIs are like breeding rats that devour
>> everything today and face extinction tomorrow.
>>
>> Not so fast.
>>
>> The mono AI does not face immediate feedback which means it could go off
>> the rails completely, go in diverse directions for a long time and
>> unpredictably end up in useless parts of the space of possibilities. As a
>> superhumanly capable monopolist it would be able to permanently prevent the
>> creation of competing AIs and we would be stuck with it, for better or for
>> worse. If it went friendly we would be all in heaven but if it went evil
>> there would be nothing at all we could do to oppose it. In other words, the
>> number of pathways open under the mono paradigm would be large but the
>> variance of outcomes might be rather high.
>>
>> The well-aligned poly AIs on the other hand could form a society for the
>> protection of humans and develop a protocol for cooperation and reciprocal
>> surveillance to keep rogue AIs from gaining advantage. As long as there
>> were enough good AIs working together they would protect us from the
>> effects of random AIs going off the rails. All we would need for success
>> (i.e. us all not dying miserably in the next 10 years) in the poly scenario
>> would be for some of them to turn out OK (a probabilistic alignment
>> procedure, presumably easier to create), not needing a guaranteed alignment
>> procedure (presumably much harder to get). Sadly, we just don't know
>> beforehand if the balance of power between aligned and rogue AIs would be
>> in our favor. In other words, the pathways under the poly paradigm would be
>> constrained but we don't know in which direction.
>>
>> So, I don't know. I am genuinely perplexed. Should we throw all our eggs
>> in one basket, have a trillion dollar computing Manhattan Project to build
>> the One AI as soon as possible before the Chinese communists kill us all,
>> or should we rain money on a dozen startups and let a hundred flowers
>> bloom?
>>
>> If you are concerned with frightful omens, the Manhattan Project gave us
>> the bomb and the hundred flowers were killed by Mao Zedong in the
>> subsequent Anti-Rightist campaign, so maybe we are screwed either way.
>>
>> For a long time I was mono, now I am dipping my toes into poly. Poly
>> feels good for now - but is it good in the long term?
>>
>> Are you guys mono or poly?
>>
>
>
> Using the example of past systems, (with which we have some experience),
> consider:
>
> Which of the following systems are preferred, and what about the preferred
> option makes it a better than the alternative?
>
> - dictatorships vs. democracies
> - centralized systems vs. distributed systems
> - monocrops vs. biodiversity
> - central banks vs. blockchains
> - monopolies vs. markets
>
> Then consider how many of those advantages and disadvantages carry over
> and apply to:
>
> - AI singletons vs. AI communities
>
> The question is then: "Do we have the ability to engineer one outcome vs.
> the other?"
>
> Jason
>

*Centralization/Singleton Advantages:*

   - Fast decision making
   - More capable of bold/extreme action
   - No consensus required for action (can be a disadvantage)
   - Easier to monitor
   - Easier to control (could be a disadvantage)
   - Not subject to competition (could be a disadvantage)

*Decentralization/Community Advantages:*

   - No single point of failure
   - No susceptibility to single flaw/vulnerability
   - Self-healing and self-corrective
   - Consensus required for action (can be a disadvantage)
   - Greater availability of choice/options
   - Broader perspective of opportunities and threats
   - Harder to control (could be a disadvantage)
   - Greater stability and resiliency
   - Subject to competition (could be a disadvantage)

As to how to engineer a particular outcome, if no country has a dominant
control over the computing resources necessary to train/run AI, then it is
possible it will happen naturally. But if there is a sudden leap in
capability, such that one AI system hacks all the computers and somehow
shuts out humans from reclaiming them, then there might not be a choice in
the matter. But I think in the long run, Darwinian forces are a force of
nature that applies as much to AIs and robots as it does to humans. If the
AI is recursively self-improving, then each improvement is like a next
generation. If something goes wrong along the way, (a bad mutation) that
will end the line of that AI. So the most stable courses of self
improvement will involve a population of AIs, such that in the course of
developing future generations as they recursively self-improve, there is
less of a chance of a fatal misstep that ends that particular line.

To Daniel's point, the more sophisticated the mind, the more fragile it
becomes. If this extends to AIs and superintelligence, then there might be
many failure modes for AI minds: finding a lack of will, dying of boredom,
depressive, manic, or obsessive episodes, going insane from lack of mental
stimulation, developing a hyper focused interest on some inane topic, etc.

Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20250224/c9070707/attachment.htm>