Poxy old computers (was RE: [extropy-chat] SPACE: Spirit Problems)

[Adrian Tymes]

--- Anders Sandberg <asa at nada.kth.se> wrote:
> Each step seems to be a letting go. From handwritten
> machine code to
> assembler that is compiled, to high-level languages,
> to software living in
> an operating system, linking with other people's
> code, computers and data,
> and then having other systems generate code and the
> code change itself.
> Each time we give up control and certainty for
> flexibility and ability.

Which runs counter to commercial software development.
Who's going to QA the beast?  Who's going to certify
its utility for certain services it's sold to perform?

I know, the obvious rejoinder is, "So it won't be
useful for commercial purposes by our current
standards."  People express outrage as it is that
Windows is shipped with so many bugs.  What, then, of
the 2-year-old-child equivalent AI that can be trained
to do certain tasks like a savant with lots of memory
and several times human cognition speed, but is known
to easily fall into highly agitated states where it
produces no useful work and tries to make things
unpleasant for anyone nearby (codenamed
"tempertantrums")?

One possible answer might be to take the same approach
to quality assurance.  Give an expert system (say, a
"Yes/No Neural-Administration Net") a set of generic
bounds on acceptable behavior, plus another set
specific to a project, then let it consider likely
potential behaviors.  More bounds can be added as the
system gains experience (even those that might not be
applicable, i.e., "do not kill a human being unless
this project specifically overrides" would
automatically pass for a simple calculator).
Regardless, the QA AI then thinks up many test cases
to ferret out different types of potential
misbehaviors, and runs 'em.  Independent quality
assurance labs would have their own systems with their
own knowledge bases (and thus likely to be free from
rules like, "killing someone is okay so long as the
system's manufacturer profits from it").