I just read this evaluation of a multiple-choice test. The second question of the test is a simple yes/no question which can obviously be guessed correctly with a certain probability. The analysis has to cope with the fact that an (unknown) fraction g of the participants simply guessed. The most interesting figure is what fraction of the participants k knowingly checked the correct answer, but this figure is unknown due to the unknown number of guessers. Instead, the fraction of correct answers / total no. of answers a is known.

Let's assume that if guessing participants pick answers uniformly at random (i.e., ignore psychologically induced biases).

The author attempts to extract k from a by visually subtracting the 'guessing score' of 0.5:

enter image description here

Intuition tells me this visualization is inappropriate and leads to an evaluation that is too pessimistic, but I can't quite tell why. Is there a good explanation of what's wrong with this picture? What would be a better visualisation of k, given that g is unknown?


Here is a more detailed plot of the results grouped by four sub-populations:

enter image description here

As you can see, in the "Oracle" subpopulation, merely 47.7% answered correctly (53.6% - 5.9%). Therefore, it is most likely the case that everyone either a) knows the correct answer or b) guesses uniformly at random, since the "Oracle" group is below the guessing threshold. Rather, it is possible for participants to be convinced about the wrong answer.

  • $\begingroup$ I don't understand your second graph (that you just added). $\endgroup$ – Peter Flom - Reinstate Monica Mar 18 '14 at 20:43
  • $\begingroup$ @PeterFlom: I edited my description, is it clear now? $\endgroup$ – blubb Mar 18 '14 at 20:48

If we assume that everyone either a) knew the right answer or b) Guessed completely randomly then we can estimate the proportion in a):

Find the proportion who got the wrong answer, double that, subtract it from 1 and voila! In your diagram it looks like 46.4% got the wrong answer; double that is 92.8; 100-92.8 = 7.2% knew the answer.

  • $\begingroup$ Your assumption translates to 1 = k + g. In this case, a = k + 1/2*g, which resolves to k = 1-2a. That explains the diagram's underlying assumption. However, the assumption must be wrong (see my edit to the question). Given this, what is a better visualization? $\endgroup$ – blubb Mar 18 '14 at 20:33
  • $\begingroup$ No, that doesn't mean my assumption is wrong; it could be that this particular group got unlucky. However, my assumption likely is wrong, but there is no way to know. $\endgroup$ – Peter Flom - Reinstate Monica Mar 18 '14 at 20:51
  • $\begingroup$ You are right. I checked, and the deviation is not statistically significant (under some further assumptions of course). $\endgroup$ – blubb Mar 18 '14 at 20:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.