[ExI] standard form for creating a test

[Anders Sandberg]

Here is my take on it. Start with Bayesian statistics, end up with a 
fairly simple algorithm:

Let us use Bayes theorem:

P(skill|answers)=P(answers|skill)P(skill)/P(answers)

=K P(a1,a2,a3,...|skill)P(skill) [K is a normalization factor; we only 
care about relative differences of skill, not the absolute probability]

=K P(a1|skill)P(a2|skill)...P(skill) [assume independent answers]

=K P(a1|skill)P(a2|skill)...P(skill) [assume uniform prior]

So if we get a sequence to answers we can calculate the likeliehood of 
different skill levels. Let's assume the questions have different 
difficulties i. Then the probability for a particular question that a 
person with skill j will be:

P(right answer difficulty i| skill=j) = C sigmoid(j-i)
P(wrong i|j)=1-C sigmoid(j-1)

where C is a normalization and sigmoid represents some suitable 
monotonic sigmoid function like .5+.5*tanh(x). In fact, I think we can 
use a step function like .5+.5*sign(x) - people better than the question 
difficulty are assumed to always answer right.

So this gives us an algorithm:

Assume we have N skill levels
Set P(j) = 1/N
while (not done)
    ask question of difficulty i
    for j=1:N
       if (right answer)
          P(j) = P(j)* sigmoid(j-i)
       else
          P(j)=P(j)*(1-sigmoid(j-i))
    end
    normalise P
end

The final issue is how to select the question. We want to get as much 
information as possible, so a good method would be to ask the question 
where an answer maximises the change of sum P(j) log(P(j)), the 
entropy/uncertainty about the skill. I don't have the time to calculate 
the right solution, but it would not be hard to either loop over j to 
find the right difficulty i, or to use the heuristic to ask a question 
that is close to the expectation of j: i = round(sum j*P(j))

There you are.

-- 
Anders Sandberg,
Future of Humanity Institute
Philosophy Faculty of Oxford University