[ExI] Unfrendly AI is a mistaken idea.

[Eugen Leitl]

On Tue, Jun 12, 2007 at 02:58:58PM +0400, Vladimir Nesov wrote:

> You can define restricted subset of programs with tractable behaviour and
> implement you system in that subset. It's just diffucult in practice, as it takes

But you operate purely in the emergent effect domain.
A program is made from very simple components (instructions)
which have no behaviour in itself.

It's the sum of it that is doing useful/interesting, and
frequently unanticipated things.

> many times over in work, training on the level you can't supply in
> large quantities, and slower resulting code. And it probably can't be
> usefully applied to complicated AI (as too much is in unforeseen data, and
> assertions you want to check against can't be formulated).

Precisely. Formal system verification can't scale beyond trivial
complexity levels. Formal system verification is absolutely 
useless in real-world AI, unless you're operating on the formal
domain to start with.