[extropy-chat] The nature of SI

[Hal Finney]

Super-intelligence (SI) is a confusing and somewhat paradoxical
notion.  At one extreme it may be seen as just a sped-up form of human
intelligence.  At the other it may be seen as an almost mystical ability
to know without limits.  Here's what I think.

Our understanding of the nature of intelligence is bound up in our
understanding of humanity.  We are the smartest things around, although
our computers are beginning to exceed our abilities in some limited areas.
But human minds are not good models for abstract intelligence.  We have
many specific limitations and adaptations for the needs of living and
survival.  And we are bound up in an evolutionary framework which gives
us goals and drives that guide and control our intelligent abilities.

Our minds, while versatile, are actually quite limited in their operation.
We can't easily think of more than one or two abstract thoughts at a time.
We can't hold more than 10 or so distinct items in short term memory.
It's very difficult to hold a large and complex structure in the mind
and keep track of it.  A SI should be able to transcend these kinds of
limitations, with sufficient resources.  It can think of many things
at once, and can work on large numbers of data items simultaneously.
Making these architectural changes would be a step towards SI without
even needing to speed up or otherwise enhance the mind.

Another property of human minds is that they are designed to solve the
problems of survival and reproduction.  Everything about life is oriented
towards these goals.  It often seems to us that these concepts are an
inherent part of intelligence, that an SI would automatically share these
goals.  But I don't think that is the case.  Intelligence has nothing
to do with survival per se.  It's perfectly reasonable to imagine an AI
which is uninterested in survival or reproduction but just works in some
specific problem domain.  Expanding the domain does not automatically
add a survival goal.

Stripped of these human-oriented features, intelligence is a general and
abstract ability to solve problems.  I think of intelligence as a black
box where you put in a goal and information about the world, and out
comes a recommended course of action.  In this context, an organism or
actor, whether biological or artificial, is more than its intelligence.
It's somewhat misleading to speak of an SI as an entity.  The intelligence
itself is just part of the picture.  There also need to be other parts:
a perceptual part to gain information about the world; a goal-supplying
part to say what to try to accomplish; and an acting part to make use
of the outputs from the intelligence.

All of these parts are present in humans and other organisms.  We have
perceptual organs to learn about the world, plus instinctive knowledge
built into our genome; we have goals supplied to us by evolution; and
we can act by moving and speaking, to implement the plans and decisions
our brains create.  The same thing will have to be true of an SI: it will
have to be given knowledge and perception about the world; it will need
goals that it should try to accomplish; and it needs to have a way to
communicate or act on the results of its thinking.

This is all uncontroversial except perhaps for the goals.  There have been
some suggestions that an SI will decide its own goals.  This doesn't make
sense in my model.  Intelligence is a problem solving capability, and it
needs a problem to solve.  That problem is defined by the goals and the
data.  Intelligence by itself cannot come up with either.

In this conception, what is the nature of intelligence?  In abstract,
intelligent analysis is a matter of search and optimization.  The
intelligence searches through an abstract space of possible actions,
uses data about the world to determine consequences, and ranks those
consequences against its goals to determine the optimal action.
All problems of intelligence can be put into this abstract framework.

However, this does not imply that the actual implementation of an SI is
a simple brute force search.  This is merely a statement of the nature
of the problem faced by the SI.  The actual tactics used to solve it
are almost infinitely variable, and its ability to discover, choose and
exploit advanced techniques for improving the search are what make the SI
"super".

I find the model employed by Juergen Schmidhuber described at
http://www.idsia.ch/~juergen/oops.html to be helpful in understanding how
a general problem solver can work.  The idea is that the SI has a strategy
for solving the problem, which can be thought of as a specific program
algorithm.  It includes problem-specific knowledge, heuristic tricks for
shortening the search, all kinds of tricks and clever insights to make
the analysis go faster or be more accurate.  But the SI doesn't just sit
there and think about the problem using this strategy.  In addition,
it also spends time looking for better strategies, better heuristics,
better ways to short-circuit some of the analysis, new patterns and
insights that it can exploit.  A certain percentage of its resources
are devoted to the straight problem solving, and another percentage are
spent trying to improve the problem solving technique.

But (and this is the key insight, which goes back at least to Levin in
the 1970s), this secondary effort of improving its strategy is itself
a matter of problem solving: of search and optimization.  And that's
exactly what the SI is designed to do.  It can use the same abstract
techniques to solve this problem that it uses to solve the base problem.
It could use a relatively simple approach of trying out alterations
to the strategy for the base search; or it can have more complex and
effective methods to improve its base strategy.

So we have the base strategy to solve the problem; and we have the second
order strategy to find ways to improve the base strategy.  It should be
clear that it does not stop there.  There is a third order problem of
improving the second order one, and so on, forever.  Levin showed, for
a specific class of problems, a simple method for the AI to distribute
its resources over the base problem and all the higher order problems,
such that in the end the problem would be solved within a constant factor
of as fast as if the theoretically optimal strategy had been used from
the beginning.  Schmidhuber and his students have extended Levin's
results to a larger class of problems.

I believe that this general strategy, of solving problems, and solving
meta-problems and meta-meta-problems and so on, is the right way to
think of the operation of an SI.  And it looks to me like the work
in this field of "universal search" is pretty close to laying out the
blueprint for the optimal architecture of an SI, an engine which will
solve any problem as fast as possible given its resources.

Now, in practice, we will want to give any AI a head start, by building in
world-knowledge and our best guesses at optimal strategies at each level.
We have billions of years of knowledge coded in our genes, and millennia
of cultural progress in our minds, and we don't want to make the SI
have to re-discover all this from scratch.  But once we have done our
part, the SI will operate based on the universal search architecture.
It will self-improve, revise and optimize its strategies at each level
of analysis.

This picture of the nature of SI lies somewhere between the extremes
I described above.  It is far more than a human intelligence, as it has
virtually unlimited ability to revise its strategies at any level.  But it
is far less than omniscient, as it constantly works with approximations
and divides up its resources among the many simultaneous problems it is
trying to solve.

Hal