I just read this evaluation of a multiple-choice test. The second question of the test is a simple yes/no question which can obviously be guessed correctly with a certain probability. The analysis has to cope with the fact that an (unknown) fraction
g of the participants simply guessed. The most interesting figure is what fraction of the participants
k knowingly checked the correct answer, but this figure is unknown due to the unknown number of guessers. Instead, the fraction of correct answers / total no. of answers
a is known.
Let's assume that if guessing participants pick answers uniformly at random (i.e., ignore psychologically induced biases).
The author attempts to extract
a by visually subtracting the 'guessing score' of 0.5:
Intuition tells me this visualization is inappropriate and leads to an evaluation that is too pessimistic, but I can't quite tell why. Is there a good explanation of what's wrong with this picture? What would be a better visualisation of
k, given that
g is unknown?
Here is a more detailed plot of the results grouped by four sub-populations:
As you can see, in the "Oracle" subpopulation, merely 47.7% answered correctly (53.6% - 5.9%). Therefore, it is most likely the case that everyone either a) knows the correct answer or b) guesses uniformly at random, since the "Oracle" group is below the guessing threshold. Rather, it is possible for participants to be convinced about the wrong answer.