[ExI] tests again

[David Lubkin]

The varied ways that someone might answer a question aren't
necessarily an insurmountable problem.

I'd been thinking about the issue as it relates to conventional test
items. The problem with a number sequence item like

	3  5  7  9 11  ___

is that, mathematically speaking, any number can come next.
Given any finite sequence, a formula can be devised that will
yield that sequence. Which turns it from math or pattern recognition
or reasoning to Family Feud. Guess the answer that the test
creator had had in mind.

The right way is probably to give both the next number and the
formula that yields the number you gave. On grading, if it's the
answer you'd expected, mark as correct. If not, look at the
formula given.

Likewise for analogy problems.

	fish : orange :: bicycle : _________

should not be a test of whether you guessed the answer the
author had in mind. Especially since these are used on IQ
tests or their academic surrogates (e.g., the Miller Analogies
Test). If you're smarter than the test's author, you're apt to
see connections that he hadn't.

So, again, the test taker should briefly explain why her
answer works. On grading, don't check the explanation
unless the expected answer wasn't selected.

(The answer I'm looking for in the analogy above is
Seder plate.)

For your situation, the varieties and sophistications of
answers you'd come up with can serve as filters to reduce
the number of answers that have to be looked at by a
human grader.

In all three situations, once a novel answer has been
confirmed — or rejected — the grading software can be
set up to handle that answer automatically in the future.

This is essentially the best strategy historically for expert
systems — guided machine learning.


-- David.