[ExI] Limiting factors of intelligence explosion speeds

[Eugen Leitl]

On Thu, Jan 20, 2011 at 05:04:01PM -0500, Richard Loosemore wrote:

>> Unnecessary for darwinian systems. The process is dumb as dirt,
>> but it's working quite well.
>
> By even uttering the phrase "darwinian systems" you introduce a raft of  
> assumptions that all have huge implications.

Duh.

> This implies a process in which the entire AGI is improving as a result  
> of it being introduced into a big ecosystem of other AGIs, all exposed  

Yes, exactly.

> to a shaping environment, and with test-and-improve mechanisms akin to  
> those that operate in nature, to evolve biological systems.

And in memetic evolution, and in the economy.

> That is a GIGANTIC set of assumptions.  At the very least the AGIs have  

Yet it is the default mode of reality.

> to be made in very large numbers, to get meaningful competition and  
> fitness pressures ..... so if each is initially the size of a  

Speaking about assumptions.

> supercomputer, this process of improvement won't even start until we  
> have the resources to build thousands or millions of AGI supercomputers!

Think the Internet 30+ years hence.

> And, also, the fitness function has to do..... what, exactly?  Make them  

What, exectly, is your fitness function you're subject to? Can you
write a formal expression for it? Or do you deny you're even
subject to selection?

> compete for mates, according to the size of their computational muscles?  
>  Obviously a stupid idea, but if not that, then what?  And when do they  

Speaking about stupid ideas, you're pretty good at these.

> get killed off, in this evolutionary competition?  Are they being bred  
> for their logical abilities, their compassion, their sensitivity...?
>
> And assuming that they do not start with fantastically  
> greater-than-human thought speed, does this AGI evolutionary process  
> require them to live for an entire human maturation period before they  
> have babies?
>
> None of this makes any sense.

Try ideas that make sense, then.

> Somehow, the idea that genetic algorithms can improve their fitness has  
> exploded out of control and become the idea that AGIs can improve by an  
> evolutionary mechanism, but without any meaningful answers to these  
> critical questions about the evolutionary process ON THAT LARGER SCALE.
>
> What works for a GA -- the millisecond generation turnaround you mention  
> below -- is completely nuts for a full-blown AGI.

A million ms is about quarter an hour. A billion ms is two weeks.

> Let me state this clearly.  If an AGI can engage in meaningful  
> interaction with the world, sufficient for something to evaluate its  

Co-evolution is mostly others, embedded in an environment.
Rendering artificial reality at equivalent speedup is cheap.

> performance and decide on its "fitness", in the space of one  
> millisecond, then you already have a superintelligent AGI!  It would  
> probably already be operating at a million times human speed (and having  
> some pretty weird interactions with the universe, at that speed), if we  
> presume it to be capable of having its fitness judged after one 
> millisecond.
>
> I hereby declare this whole "build an AGI by darwinian evolution" idea  
> to be so logically incoherent that it does not need to detain us any 
> longer.

See, you're doing it again. But since you're a product of darwinian 
evolution, you're so logically incoherent that we can disregard whatever
you say. How convenient!

>
>
>
>
> Richard Loosemore
>
>
>>> usual.  I do not believe it makes sense to talk about what happens  
>>
>> If you define the fitness function, and have ~ms generation
>> turaround it's not quite as usual anymore.
>>
>>> *before* that point as part of the "intelligence explosion".
>>>
>>> D)  When such a self-understanding system is built, it is unlikely that
>>
>> I don't think that a self-understanding system is at all possible.
>> Or, rather, it would perform better than a blind optimization.
>>
>>> it will be the creation of a lone inventor who does it in their shed at
>>> the bottom of the garden, without telling anyone.  Very few of the "lone
>>> inventor" scenarios (the Bruce Wayne scenarios) are plausible.
>>
>> I agree it's probably a large scale effort, initially.
>>
>>> E)  Most importantly, the invention of a human-level, self-understanding
>>
>> I wonder where the self-understanding meme is coming from. It's  
>> certainly pervasive enough.
>>
>>> AGI would not lead to a *subsequent* period (we can call it the
>>> "explosion period") in which the invention just sits on a shelf with
>>> nobody bothering to pick it up.  A situation in which it is just one
>>> quiet invention alongside thousands of others, unrecognized and not
>>> generally believed.
>>>
>>> F)  When the first human-level AGI is developed, it will either require
>>> a supercomputer-level of hardware resources, or it will be achievable
>>
>> Bootstrap takes many orders of mangnitude more resources than required
>> for operation. Even before optimization happens.
>>
>>> with much less.  This is significant, because world-class supercomputer
>>> hardware is not something that can quickly be duplicated on a large
>>> scale.  We could make perhaps hundreds of such machines, with a massive
>>
>> About 30 years from now, TBit/s photonic networking is the norm.
>> The separation between core and edge is gone, and inshallah, so
>> well policy enforcement. Every city block is a supercomputer, then.
>>
>>> effort, but probably not a million of them in a couple of years.
>>
>> There are a lot of very large datacenters with excellent network
>> cross-section, even if you disregard large screen TVs and game  
>> consoles on >GBit/s residential networks.
>>
>>> G)  There are two types of intelligence speedup:  one due to faster
>>> operation of an intelligent system (clock speed) and one due to
>>
>> Clocks don't scale, eventually you'll settle for local asynchronous,
>> with large-scale loosely coupled oscillators synchronizing.
>>
>>> improvment in the type of mechanisms that implement the thought
>>> processes.  Obviously both could occur at once, but the latter is far
>>
>> How much random biochemistry tweaking would improve dramatically
>> on the current CNS performance? As a good guess, none. So once you've
>> reimplemented the near-optimal substrate, dramatic improvements
>> are over. This isn't software, this is direct implmenentation of
>> neural computational substrate in as thin hardware layer this
>> universe allows us.
>>
>>> more difficult to achieve, and may be subject to fundamental limits that
>>> we do not understand.  Speeding up the hardware, on the other hand, has
>>
>> I disagree, the limits are that of computational physics, and these are
>> fundamentally simple.
>>
>>> been going on for a long time and is more mundane and reliable.  Notice
>>> that both routes lead to greater "intelligence", because even a human
>>> level of thinking and creativity would be more effective if it were
>>> happening (say) a thousand times faster than it does now.
>>
>> Run a dog for a gigayear, still no general relativity.
>>
>>> *********************************************
>>>
>>> Now the specific factors you list.
>>>
>>> 1) Economic growth rate
>>>
>>> One consequence of the above reasoning is that economic growth rate
>>> would be irrelevant.  If an AGI were that smart, it would already be
>>
>> Any technology allowing you to keep a mind in a box will allow you
>> to make a pretty good general assembler. The limits of such technology
>> are energy and matter fluxes. Buying and shipping widgets is only a
>> constraining factor in the physical layer bootstrap (if at all necessary,
>> 30 years hence all-purpose fabrication has a pretty small footprint).
>>
>>> obvious to many that this was a critically important technology, and no
>>> effort would be spared to improve the AGI "before the other side 
>>> does".  Entire national economies would be sublimated to the goal of 
>>> developing  the first superintelligent machine.
>>
>> This would be fun to watch.
>>
>>> In fact, economic growth rate would be *defined* by the intelligence
>>> explosion projects taking place around the world.
>>>
>>>
>>> 2) Investment availability
>>>
>>> The above reasoning also applies to this case.  Investment would be   
>>> irrelevant because the players would either be governments or 
>>> frenzied
>>> bubble-investors, and they would be pumping it in as fast as money 
>>> could  be printed.
>>>
>>>
>>> 3) Gathering of empirical information (experimentation, interacting with
>>> an environment).
>>>
>>> So, this is about the fact that the AGI would need to do some   
>>> experimentation and interaction with the environment.  For example, 
>>> if  
>>
>> If you have enough crunch to run a mind, you have enough crunch to
>> run really really really good really fast models of the universe.
>>
>>> it wanted to reimplement itself on faster hardware (the quickest 
>>> route  to an intelligence increase) it would probably have to set up 
>>> its own  hardware research laboratory and gather new scientific data 
>>> by doing  experiments, some of which would go at their own speed.
>>
>> You're thinking like a human.
>>
>>> The question is:  how much of the research can be sped up by throwing
>>> large amounts of intelligence at it?  This is the parallel-vs-serial
>>> problem (i.e. you can't make a baby nine times quicker by asking nine 
>>>  women to be pregnant for one month).
>>
>> It's a good question. I have a hunch (no proof, nothing) that the
>> current way of doing reality modelling is extremely inefficient.
>> Currently, experimenters have every reason to sneer at modelers.
>> Currently.
>>
>>> This is not a factor that I believe we can understand very well ahead of
>>> time, because some experiments that look as though they require
>>> fundamentally slow physical processes -- like waiting for a silicon
>>> crystal to grow, so we can study a chip fabrication mechanism -- may
>>> actually be dependent on smartness, in ways that we cannot anticipate.
>>> It could be that instead of waiting for the chips to grow at their own
>>> speed, the AGI can do clever micro-experiments that give the same
>>> information faster.
>>
>> Any intelligence worth its salt would see that it would use computational
>> chemistry to bootstrap molecular manufacturing. The grapes could be hanging
>> pretty low there.
>>
>>> This factor invites unbridled speculation and opinion, to such an extent
>>> that there are more opinions than facts.  However, we can make one
>>> observation that cuts through the arguments.  Of all the factors that 
>>>  determine how fast empirical scientific research can be carried out, 
>>> we  know that intelligence and thinking speed of the scientist 
>>> themselves  *must* be one of the most important, today.  It seems 
>>> likely that in our  present state of technological sophistication, 
>>> advanced research  projects are limited by the availability and cost 
>>> of intelligent and  experienced scientists.
>>
>> You can also vastly speed up the rate of prototyping by scaling down
>> and proper tooling. You see first hints of that in lab automation,
>> particularly microfluidics. Add ability to fork off dedicated investigators
>> at the drop of a hat, and things start happening, and in a positive-feedback
>> loop.
>>
>>> But if research labs around the world have stopped throwing *more*
>>> scientists at problems they want to solve, because the latter cannot be
>>> had, or are too expensive, would it be likely that the same research
>>> labs ar *also*, quite independently, at the limit for the physical rate
>>> at which experiments can be carried out?  It seems very unlikely that 
>>>  both of these limits have been reached at the same time, because 
>>> they  cannot be independently maximized.  (This is consistent with 
>>> anecdotal  reports:  companies complain that research staff cost a 
>>> lot, and that  scientists are in short supply:  they don't complain 
>>> that nature is just  too slow).
>>
>> Most monkeys rarely complain that they're monkeys. (Resident monkeys
>> excluded, of course).
>>
>>> In that case, we should expect that any experiment-speed limits lie up
>>> the road, out of sight.  We have not reached them yet.
>>
>> I, a mere monkey, can easily imagine two orders of magnitude speed
>> improvements. Which, of course, result in a positive autofeedback loop.
>>
>>> So, for that reason, we cannot speculate about exactly where those
>>> limits are.  (And, to reiterate:  we are talking about the limits that
>>> hit us when we can no longer do an end-run around slow experiments by
>>
>> I do not think you will need slow experiments. Not slow by our standards,
>> at least.
>>
>>> using our wits to invent different, quicker experiments that give the
>>> same information).
>>>
>>> Overall, I think that we do not have concrete reasons to believe that
>>> this will be a fundamental limit that stops the intelligence explosion
>>> from taking an AGI from H to (say) 1,000 H.  Increases in speed within
>>> that range (for computer hardware, for example) are already expected,
>>> even without large numbers of AGI systems helping out, so it would 
>>> seem  to me that physical limits, by themselves, would not stop an 
>>> explosion  that went from I = H to I = 1,000 H.
>>
>> Speed limits (assuming classical computation) do not begin to take hold
>> before 10^6, and maybe even 10^9 (this is more difficult, and I do not
>> have a good model of wetware at 10^9 speedup to current wallclock).
>>
>>> 4)  Software complexity
>>>
>>> By this I assume you mean the complexity of the software that an AGI
>>> must develop in order to explode its intelligence.  The premise is
>>> that even an AGI with self-knowledge finds it hard to cope with the
>>> fabulous complexity of the problem of improving its own software.
>>
>> Software, that's pretty steampunk of you.
>>
>>> This seems implausible as a limiting factor, because the AGI could
>>> always leave the software alone and develop faster hardware.  So long as
>>
>> There is no difference between hardware and software (state) as far
>> as advanced cognition is concerned. Once you've covered the easy
>> gains in first giant co-evolution steps further increases are much
>> more modest, and much more expensive.
>>
>>> the AGI can find a substrate that gives it (say) 1,000 H thinking-speed,
>>
>> We should be able to do 10^3 with current technology.
>>
>>> we have the possibility for a significant intelligence explosion.
>>
>> Yeah, verily.
>>
>>> Arguing that software complexity will stop the initial human level 
>>> AGI  
>>
>> If it hurts, stop doing it.
>>
>>> from being built is a different matter.  It may stop an intelligence  
>>> explosion from happening by stopping the precursor events, but I take 
>>>  that to be a different type of question.
>>>
>>>
>>> 5)  Hardware demands vs. available hardvare
>>>
>>> I have already mentioned, above, that a lot depends on whether the 
>>> first  AGI requires a large (world-class) supercomputer, or whether 
>>> it can be  done on something much smaller.
>>
>> Current supercomputers are basically consumer devices or embeddeds on  
>> steroids, networked on a large scale.
>>
>>> This may limit the initial speed of the explosion, because one of the 
>>>  critical factors would be the sheer number of copies of the AGI that 
>>> can  
>>
>> Unless the next 30 years won't see the same development as the last ones,
>> then substrate is the least of your worries.
>>
>>> be created.  Why is this a critical factor?  Because the ability to 
>>> copy  the intelligence of a fully developed, experienced AGI is one 
>>> of the big  new factors that makes the intelligence explosion what it 
>>> is:  you  cannot do this for humans, so human geniuses have to be 
>>> rebuilt from  scratch every generation.
>>>
>>> So, the initial requirement that an AGI be a supercomputer would make 
>>> it  hard to replicate the AGI on a huge scale, because the 
>>> replication rate  would (mostly) determine the 
>>> intelligence-production rate.
>>
>> Nope.
>>
>>> However, as time went on, the rate of replication would grow, as  
>>
>> Look, even now we know what we would need, but you can't buy it. But  
>> you can design it, and two weeks from now you'll get your first 
>> prototypes.
>> That's today, 30 years the prototypes might be hours away.
>>
>> And do you need prototypes to produce a minor variation on a stock
>> design? Probably not.
>>
>>> hardware costs went down at their usual rate.  This would mean that 
>>> the  *rate* of arrival of high-grade intelligence would increase in 
>>> the years  following the start of this process.  That intelligence 
>>> would then be  used to improve the design of the AGIs (at the very 
>>> least, increasing  the rate of new-and-faster-hardware production), 
>>> which would have a  positive feedback effect on the intelligence 
>>> production rate.
>>>
>>> So I would see a large-hardware requirement for the first AGI as   
>>> something that would dampen the initial stages of the explosion.  But 
>>>  
>>
>> Au contraire, this planet is made from swiss cheese. Annex at your leisure.
>>
>>> the positive feedback after that would eventually lead to an 
>>> explosion  anyway.
>>>
>>> If, on the other hand, the initial hardware requirements are modest 
>>> (as  they very well could be), the explosion would come out of the 
>>> gate at  full speed.
>>>
>>>
>>>
>>>
>>> 6)  Bandwidth
>>>
>>> Alongside the aforementioned replication of adult AGIs, which would   
>>> allow the multiplication of knowledge in ways not currently available 
>>> in  humans, there is also the fact that AGIs could communicate with 
>>> one  another using high-bandwidth channels.  This inter-AGI 
>>> bandwidth.
>>
>> Fiber is cheap. Current fiber comes in 40 or 100 GBit/s parcels.
>> 30 years hence bandwidth will be probably adequate.
>>
>>> As a separate issue, there might be bandwidth limits inside an AGI,   
>>> which might make it difficult to augment the intelligence of a single 
>>>  system.  This is intra-AGI bandwidth.
>>
>> Even now bandwidth growth is far in excess of computation growth.
>> Once you go embedded memory, you're more closely matched. But still
>> the volume/surface (you only have to communicate surface state)
>> ratio indicated the local communication is the bottleneck.
>>
>>> The first one - inter-AGI bandwidth - is probably less of an issue 
>>> for  the intelligence explosion, because there are so many research 
>>> issues  that can be split into separably-addressible components, that 
>>> I doubt we  would find AGIs sitting around with no work to do on the 
>>> intelligence  amplification project, on account of waiting for other 
>>> AGIs to get a  free channel to talk to them.
>>
>> You're making it sound so planned, and orderly.
>>
>>> Intra-AGI bandwidth is another matter entirely.  There could be   
>>> limitations on the IQ of an AGI -- for example if working memory   
>>> limitations (the magic number seven, plus or minus two) turned out to 
>>> be  caused by connectivity/bandwidth limits within the system.
>>
>> So many assumptions.
>>
>>> However, notice that such factors may not inhibit the initial phase 
>>> of  an explosion, because the clock speed, not IQ, of the AGI may be  
>>> 
>>
>> There is no clock, literally. Operations/volume, certainly.
>>
>>> improvable by several orders of magnitude before bandwidth limits 
>>> kick  in.  The reasoning behind this is the observation that neural 
>>> signal  
>>
>> Volume/surface ratio is on your side here.
>>
>>> speed is so slow.  If a brain-like system (not necessarily a whole 
>>> brain  emulation, but just something that replicated the high-level   
>>> functionality) could be built using components that kept the same 
>>> type  of processing demands, and the same signal speed.  In that kind 
>>> of  system there would then be plenty of room to develop faster 
>>> signal  speeds and increase the intelligence of the system.
>>>
>>> Overall, this is, I believe, the factor that is most likely to cause  
>>> trouble.  However, much research is needed before much can be said 
>>> with  certainty.
>>>
>>> Most importantly, this depends on *exactly* what type of AGI is being 
>>>  built.  Making naive assumptions about the design can lead to false  
>>> conclusions.
>>
>> Just think of it as a realtime simulation of a given 3d physical
>> process (higher dimensions are mapped to 3d, so they don't figure).
>> Suddenly things are simple.
>>
>>>
>>> 7)  Lightspeed lags
>>>
>>> This is not much different than bandwidth limits, in terms of the 
>>> effect  it has.  It would be a significant problem if the components 
>>> of the  machine were physically so far apart that massive amounts of 
>>> data (by  assumption) were delivered with a significant delay.
>>
>> Vacuum or glass is a FIFO, and you don't have to wait for ACKs.
>> Just fire stuff bidirectionally, and deal with transmission errors
>> by graceful degradation.
>>
>>> By itself, again, this seems unlikely to be a problem in the initial 
>>> few  orders of magnitude of the explosion.  Again, the argument 
>>> derives from  what we know about the brain.  We know that the brain's 
>>> hardware was  chosen due to biochemical constraints.  We are 
>>> carbon-based, not  silicon-and-copper-based, so, no chips in the 
>>> head, only pipes filled  with fluid and slow molecular gates in the 
>>> walls of the pipes.  But if  nature used the pipes-and-ion-channels 
>>> approach, there seems to be  plenty of scope for speedup with a 
>>> transition to silicon and copper (and  never mind all the other more 
>>> exotic computing substrates on the  horizon).  If that transition 
>>> produced a 1,000x speedup, this would be  an explosion worthy of the 
>>> name.
>>
>> Why so modest?
>>
>>> The only reason this might not happen would be if, for some reason, 
>>> the  brain is limited on two fronts simultaneously:  both by the 
>>> carbon  implementation and by the fact that bigger brains cause 
>>> disruptive  
>>
>> The brain is a slow, noisy (but one using noise to its own advantage)
>> metabolically constrained system which burns most of its metabolism  
>> for homeostasis purposes. It doesn't take a genius to sketch the
>> obvious ways in which you can reimplement that design, taking advantages
>> and removing disadvantages.
>>
>>> light-speed delays.  Or, that all non-carbon-implementation of the 
>>> brain  take us up close to the lightspeed limit before we get much of 
>>> a speedup  
>>
>> We here work with ~120 m/s, not 120 Mm/s. Reduce feature size by
>> an order of magnitude or two, and switching times of ns and ps
>> instead of ms, and c is not that big a limitation anymore.
>>
>>> over the brain.  Neither of these ideas seem plausible.  In fact, 
>>> they  both seem to me to require a coincidence of limiting factors 
>>> (two  limiting factors just happening to kick in at exactly the same 
>>> level),  which I find deeply implausible.
>>>
>>>
>>> *****************
>>>
>>> Finally, some comments about approaches to AGI that would affect the  
>>> answer to this question about the limiting factors for an 
>>> intelligence  explosion.
>>>
>>> I have argued consistently, over the last several years, that AI   
>>> research has boxed itself into a corner due to a philosophical   
>>> commitment to the power of formal systems.  Since I first started  
>>
>> Very much so.
>>
>>> arguing this case, Nassim Nicholas Taleb (The Black Swan) coined the  
>>> term "Ludic Fallacy" to describe a general form of exactly the issue 
>>> I  have been describing.
>>>
>>> I have framed this in the context of something that I called the   
>>> "complex systems problem", the details of which are not important 
>>> here,  although the conclusion is highly relevant.
>>>
>>> If the complex systems problem is real, then there is a very large 
>>> class  of AGI system designs that are (a) almost completely ignored 
>>> at the  moment, and (b) very likely to contain true intelligent 
>>> systems, and (c)  quite possibly implementable on relatively modest 
>>> hardware.  This class  
>>
>> Define "relatively modest".
>>
>>> of systems is being ignored for sociology-of-science reasons (the   
>>> current generation of AI researchers would have to abandon their 
>>> deepest  loves to be able to embrace such systems, and since they are 
>>> fallible  humans, rather than objectively perfect scientists, this is 
>>> anathema).
>>
>> Which is why blind optimization processes running on acres of
>> hardware will kick their furry little butts.
>>
>>> So, my most general answer to this question about the rate of the   
>>> intelligence explosion is that, in fact, it depends crucially on the  
>>> kind of AGI systems being considered.  If the scope is restricted to 
>>> the  current approaches, we might never actually reach human level   
>>> intelligence, and the questio is moot.
>>>
>>> But if this other class of (complex) AGI systems did start being 
>>> built,  we might find that the hardware requirements were relatively 
>>> modest  (much less than supercomputer size), and the software 
>>> complexity would  also not be that great.  As far as I can see, most 
>>> of the  
>>
>> I love this "software" thing.
>>
>>> above-mentioned limitations would not be significant within the first 
>>>  few orders of magnitude of increase.  And, the beginning of the 
>>> slope  could be in the relatively near future, rather than decades 
>>> away.
>>
>> In order to have progress, you first have to have people working on it.
>>
>>> But that, as usual, is just the opinion of an AGI researcher.  No 
>>> need  to take *that* into account in assessing the factors.  ;-)
>>
>> Speaking of AGI researchers: do you have a nice publication track of
>> yours you could dump here? 
>>
>
> _______________________________________________
> extropy-chat mailing list
> extropy-chat at lists.extropy.org
> http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat
-- 
Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org
______________________________________________________________
ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE