I'm writing a concept recognition self-test. (This is less weird than it sounds, but only slightly. Don't worry about it.) Partly to prevent people from clicking everything in sight, but mostly just to force people to think about the items harder, I have included a number of fake items on the list.
For example, suppose I ask about 25 concepts with 10 fakes. A person might choose 5 out of the 25, or 20 out of 25, or whatever. Ideally, no one would ever choose the fake items, but of course it can happen. I'm trying to work out a sound method for grading such a scheme. (This is purely for my own edification -- the scheme doesn't need to be graded since it's just a self-test -- but I'd like to see what a good method would be.)
I'm reminded of Tao's How to assign partial credit on an exam of true-false questions? which gives steep penalties for assigning high probabilities to false items. I'd like to do something conceptually similar, taking into account the number of fakes (the more the harder). I'm also open to suggestions on how many to include.
A commenter asked for an example. Let's say that I'm asking which stellar classes a person is familiar with, presenting them with the choices of class A, B, C, E. F, G, K, M, and O (these are the concepts). The person then chooses which of those 9 they know. One person might only select G, a class which includes our sun. Another person might choose each of O, B, A, F, G, K, M but leave off the unrecognized C (carbon stars) and E (this one is fake). Someone who chose E guessed on that one, but perhaps on others as well.
If I didn't include any fakes, I'd just construct a binomial confidence interval given the number of items selected out of the total. Is there a good way to modify this for my needs, or maybe something completely different?
Right now I don't have any good ideas, so anything would be welcome. My stopgap is to treat a person choosing $c$ out of $t$ total real concepts, who also chose $f$ fake concepts, as having $c$ out of $t+f$ correct, then constructing a confidence interval. So in the example, the person choosing A, B, C, and E, 3 real and 1 fake out of 8 real star classes and 1 fake, would get 3 out of 9 (average 30%, A-C 95% interval 10% to 68%) while someone who chose only A, B, and C would get 3 out of 8 (average 37.5%, A-C interval 11% to 73%). But this doesn't have a sound statistical motivation.
I'm open to take suggestions on how better to formulate my question.