[ExI] tests again

[Mike Dougherty]

On Mon, Dec 31, 2012 at 1:42 AM, spike <spike66 at att.net> wrote:

>
>
> Something cool: Wondershare Quiz Creator lets you imbed photos and video
> in your quiz.  That in itself allows a geek to write a test which nearly
> any geek would get immediately, but any non-geek could tap away on the
> internet for a week and come up blank.  For instance, a good mechanical
> technician’s question would be****
>
> **1.       ** What is this person doing?****
>
> The answer would not be multiple choice, because that would make it too
> easy.  The answer would be typed in on the keyboard: he or she is adjusting
> valves.  ****
>
> ** **
>
> Are there any mechanical geeks here who missed that one?  ****
>
>
> So now the test taker must figure out a way to write software to read the
> answers and assign a point value.****
>
> ** **
>
> All suggestions welcome, about tests, not exhaust valves or German
> engineering.****
>
>
>
It sounds like you've made test creation extremely difficult.  Each test
will have to be hand-crafted by domain experts.  I suppose you are ok with
that because it is the nature of this kind of test.  However, it will not
scale well at all.  There are likely very few technical people who also
have the patience to build tests like you describe.

Your externalized knowledge wrinkle doesn't really make the test much more
difficult to take.  I think this is also a design feature.  You want the
experts to have no issue completing the test while you weed-out the
non-experts.

You then continue to discuss how difficult this test will be to grade.  By
the time you've written an expert system to grade this test you won't need
engineers because you'll be selling your test grader.  So I feel like
you've rediscovered a basic problem for every schoolteacher who has the
fortitude to give essay-answer tests:  they have to read, understand,
assess the quality of the answers - multiplied times the number of
students.  I never appreciated that an essay test that 25 students spent
45-90 minutes answering would amount to 3-5 hours work for the teacher.  In
order to minimize the multiplier, maybe you only give your fancy test to
those few candidates who have passed initial screening by HR?

Now I think the solution is to also externalize the grading.  The taker is
presumably going to work with a team, yeah?  Why not involve the team in
assessing the accuracy of the answers?  While "adjusting valves" may be the
technical answer you need for "correct" it seems like a flip-answer.  On
the other hand, your thought process about a scaled answer smells to me
like beating around the bush hoping the teacher will be lenient.  Beyond
the successful fill-in-the-blank, the answer is subjective - so it makes
sense to have the team assessing the candidate.

With that in mind, the "solver" mechanism needs to be a framework for
crowd-sourcing the review process.  Even if the crowd is only your team of
6, they represent a deeper understanding/appreciation of the test
environment than a single manager.  If there exists professional networks
(or unions) there might be larger crowds of trusted/trustworthy reviewers.

and by "point value" you meant a multi-dimensional vector matrix of
correctness and appropriateness of answer?  :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20121231/4ec4d037/attachment.html>