3
$\begingroup$

I'm writing a concept recognition self-test. (This is less weird than it sounds, but only slightly. Don't worry about it.) Partly to prevent people from clicking everything in sight, but mostly just to force people to think about the items harder, I have included a number of fake items on the list.

For example, suppose I ask about 25 concepts with 10 fakes. A person might choose 5 out of the 25, or 20 out of 25, or whatever. Ideally, no one would ever choose the fake items, but of course it can happen. I'm trying to work out a sound method for grading such a scheme. (This is purely for my own edification -- the scheme doesn't need to be graded since it's just a self-test -- but I'd like to see what a good method would be.)

I'm reminded of Tao's How to assign partial credit on an exam of true-false questions? which gives steep penalties for assigning high probabilities to false items. I'd like to do something conceptually similar, taking into account the number of fakes (the more the harder). I'm also open to suggestions on how many to include.

A commenter asked for an example. Let's say that I'm asking which stellar classes a person is familiar with, presenting them with the choices of class A, B, C, E. F, G, K, M, and O (these are the concepts). The person then chooses which of those 9 they know. One person might only select G, a class which includes our sun. Another person might choose each of O, B, A, F, G, K, M but leave off the unrecognized C (carbon stars) and E (this one is fake). Someone who chose E guessed on that one, but perhaps on others as well.

If I didn't include any fakes, I'd just construct a binomial confidence interval given the number of items selected out of the total. Is there a good way to modify this for my needs, or maybe something completely different?

Right now I don't have any good ideas, so anything would be welcome. My stopgap is to treat a person choosing $c$ out of $t$ total real concepts, who also chose $f$ fake concepts, as having $c$ out of $t+f$ correct, then constructing a confidence interval. So in the example, the person choosing A, B, C, and E, 3 real and 1 fake out of 8 real star classes and 1 fake, would get 3 out of 9 (average 30%, A-C 95% interval 10% to 68%) while someone who chose only A, B, and C would get 3 out of 8 (average 37.5%, A-C interval 11% to 73%). But this doesn't have a sound statistical motivation.

I'm open to take suggestions on how better to formulate my question.

Improve this question
$\endgroup$
12
  • $\begingroup$ Could you explain what a "fake item" might be? $\endgroup$
    – whuber ♦
    Aug 23, 2016 at 17:40
  • $\begingroup$ @whuber If I were asking about stars, it might be "Class E". If I were talking about programming languages it might be A++. If I were talking about linguistics it might be the Djerashan language family. Does that help? If you have a better term I'd be happy to use it. $\endgroup$
    – Charles
    Aug 23, 2016 at 18:18
  • 1
    $\begingroup$ OK, it appears to be a particular kind of incorrect option--or maybe it's a nonsensical prompt? What are the test prompts and the possible responses? Are they multiple-choice questions? Can there be more than one correct choice? What's the purpose of grading? Is a grade a single number or possibly something more complex? $\endgroup$
    – whuber ♦
    Aug 23, 2016 at 18:22
  • 1
    $\begingroup$ @whuber Let's use stars as an example. I might ask which stellar classes a person is familiar with, presenting them with the choices of class A, B, C, E. F, G, K, M, and O. All they do (I know this sounds funny, but this is what I'm doing for reasons that don't matter here) is choose which of those 9 they know. A person might only select G, a class which includes our sun. Another person might choose each of O, B, A, F, G, K, M but leave off the unrecognized C (carbon stars) and E (this one is fake). Someone who chose E guessed on that one, but perhaps on others as well. $\endgroup$
    – Charles
    Aug 23, 2016 at 18:28
  • 1
    $\begingroup$ @whuber I don't know if this will illuminate, but this is the sort of thing I'm thinking of, where several of the choices are fake. $\endgroup$
    – Charles
    Aug 23, 2016 at 22:07

1 Answer 1

Reset to default
0
$\begingroup$

I reframe: You have 8 true/false questions. For 7 the correct answer is "true" and for one the correct answer is "false". The user/testt/subject can commit two types of errors: including an item that should have been excluded, and vice versa.

You COULD treat erroneous inclusion and erroneous exclusion as equally "bad". So, in your example, 3 correct and 5 incorrect (one erroneous inclusion and four erroneous exclusions).

Or you could treat them differently. Then you must compare (3,1,4) to the ideal (7,0,0) somehow.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.