<div class="gmail_quote">On Mon, Dec 31, 2012 at 1:42 AM, spike <span dir="ltr"><<a href="mailto:spike66@att.net" target="_blank">spike66@att.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div link="blue" vlink="purple" lang="EN-US"><div><p class="MsoNormal"><span style="color:#1f497d"> </span></p><p class="MsoNormal"><span style="color:#1f497d">Something cool: Wondershare Quiz Creator lets you imbed photos and video in your quiz. That in itself allows a geek to write a test which nearly any geek would get immediately, but any non-geek could tap away on the internet for a week and come up blank. For instance, a good mechanical technician’s question would be<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"></span></p><p><u></u><span style="color:#1f497d"><span>1.<span style="font:7.0pt "Times New Roman""> </span></span></span><u></u><span style="color:#1f497d"> What is this person doing?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"></span></p><p class="MsoNormal"><span style="color:#1f497d">The answer would not be multiple choice, because that would make it too easy. The answer would be typed in on the keyboard: he or she is adjusting valves. <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="color:#1f497d">Are there any mechanical geeks here who missed that one? <u></u><u></u></span></p><p class="MsoNormal">
<span style="color:#1f497d"></span></p><br><p class="MsoNormal"><span style="color:#1f497d">So now the test taker must figure out a way to write software to read the answers and assign a point value.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="color:#1f497d">All suggestions welcome, about tests, not exhaust valves or German engineering.<span class="HOEnZb"><font color="#888888"><u></u><u></u></font></span></span></p>
<span class="HOEnZb"><font color="#888888"><p class="MsoNormal"><span style="color:#1f497d"></span><br></p></font></span></div></div></blockquote></div><br>It sounds like you've made test creation extremely difficult. Each test will have to be hand-crafted by domain experts. I suppose you are ok with that because it is the nature of this kind of test. However, it will not scale well at all. There are likely very few technical people who also have the patience to build tests like you describe.<br>
<br>Your externalized knowledge wrinkle doesn't really make the test much more difficult to take. I think this is also a design feature. You want the experts to have no issue completing the test while you weed-out the non-experts.<br>
<br>You then continue to discuss how difficult this test will be to grade. By the time you've written an expert system to grade this test you won't need engineers because you'll be selling your test grader. So I feel like you've rediscovered a basic problem for every schoolteacher who has the fortitude to give essay-answer tests: they have to read, understand, assess the quality of the answers - multiplied times the number of students. I never appreciated that an essay test that 25 students spent 45-90 minutes answering would amount to 3-5 hours work for the teacher. In order to minimize the multiplier, maybe you only give your fancy test to those few candidates who have passed initial screening by HR?<br>
<br>Now I think the solution is to also externalize the grading. The taker is presumably going to work with a team, yeah? Why not involve the team in assessing the accuracy of the answers? While "adjusting valves" may be the technical answer you need for "correct" it seems like a flip-answer. On the other hand, your thought process about a scaled answer smells to me like beating around the bush hoping the teacher will be lenient. Beyond the successful fill-in-the-blank, the answer is subjective - so it makes sense to have the team assessing the candidate.<br>
<br>With that in mind, the "solver" mechanism needs to be a framework for crowd-sourcing the review process. Even if the crowd is only your team of 6, they represent a deeper understanding/appreciation of the test environment than a single manager. If there exists professional networks (or unions) there might be larger crowds of trusted/trustworthy reviewers. <br>
<br>and by "point value" you meant a multi-dimensional vector matrix of correctness and appropriateness of answer? :)<br><br><br>