# What will be the correct answer, if we modify the “Best statistics question ever”?

There is a popular question, called "Best statistics question ever".

If you choose an answer to this question at random, what is the chance you will be correct?

A) 25% B) 50% C) 60% D) 25%

This task is not very difficult, the correct answer is 0%. But if we modify it like this:

If you choose an answer to this question at random, what is the chance you will be correct?

A) 50% B) 25% C) 60% D) 50%

What will be the correct answer? Do we have two correct answers: 25% and 50%, or there is no correct answer, as with this two correct answers the chance of choosing the correct answer is in fact 75% (but we do not have 75% written on the desk)?

By the way. Does the answer 0% remains correct answer, the third correct answer in this case?

-
The answer, of course, depends on how the random choice is made. "At random" does not always mean "uniformly at random"... ;) –  MånsT Jun 12 '12 at 18:10
Lets assume "uniformely at random". Logic behind the original question was:As an answer 25% has 50% probability to be choosen, and an answers 50% and 60% have 25% probability to be choosen, this answers are not correct. The answer 0% is correct, as probability to choose it is 0%. –  Nick Jun 12 '12 at 18:32
This problem was exhaustively analyzed on math.SE awhile back. –  cardinal Jun 12 '12 at 18:32
@cardinal Could you provide a link or some hints for how to search for the discussion on math.SE? E.g. "+best +statistics" doesn't get me anywhere. –  whuber Jun 12 '12 at 18:54
@whuber: Here's the one with the most votes. There were others, but maybe they got closed/merged. –  cardinal Jun 12 '12 at 19:08
show 1 more comment

The apparent paradoxes (of logic or probability) can be resolved by framing the questions clearly and carefully.

The following analysis is motivated by the idea of defending an answer: when a test-taker can exhibit a possible state of affairs (consistent with all available information) in which their answer indeed is correct, then it should be marked as correct. Equivalently, an answer is incorrect when no such defense exists; it is considered correct otherwise. This models the usual interactions between (benevolent, rational) graders and (rational) test-takers :-). The apparent paradox is resolved by exhibiting multiple such defenses for the second question, only one of which could apply in any instance.

I will take the meaning of "random" in these questions in a conventional sense: to model a random choice of answer, I will write each answer on a slip of paper ("ticket") and put it in a box: that will be four tickets total. Drawing a ticket out of the box (after carefully and blindly shuffling the box's contents) is a physical model for a "random" choice. It motivates and justifies a corresponding probability model.

Now, what does it mean to "be correct"? In my ignorance, I will explore all possibilities. In any case, I take it as definite that zero, one, or even more of the tickets may be "correct." (How might I know? I simply consult the grading sheet!) I will mark the "correct" answers as such by writing the value $1$ on each correct ticket and writing $0$ on the others. That's routine and should not be controversial.

An obvious but important thing to notice is that the rule for writing $0$ or $1$ must be based solely on the answer written on each ticket: mathematically, it is a mapping (or reassignment) sending the set of listed answers ($\{.25, .50, .60\}$ in both questions) into the set $\{0,1\}$. This rule is needed for self-consistency.

Let's turn to the probabilistic element of the question: by definition, the chance of being correct, under a random drawing of tickets, is the expectation of the values with which they have been marked. The expectation is computed by summing the values on the tickets and dividing that by their total number. It will therefore be either $0$, $.25$, $.50$, $.75$, or $1$.

A marking will make sense provided that only the tickets whose answers equal the expectation are marked with $1$s. This also is a self-consistency requirement. I claim that this is the crux of the matter: to find and interpret the markings that make sense. If there are none, then the question itself can be branded as being meaningless. It there is a unique marking, then there will be no controversy. Only if two or more markings make sense will there be any potential difficulty.

Which markings make sense?

We don't even need to make an exhaustive search. In the first question, the expectations listed on the tickets are 25%, 50% and 60%. The latter is impossible with four tickets. The first would require exactly one ticket to be marked; the second, two tickets. That gives at most $3+3=6$ possible markings to explore. The only marking that makes sense puts $0$s on each ticket. For this marking, the expectation is $(0+0+0+0)/4 = 0$. That justifies the stated answer to the first question. (Arguably, the sole correct response to the first question is not to select any answer!)

In the second question, the same answers appear and once again there are six markings to explore. This time, three markings are self-consistent. I tabulate them:

Solution 1                Solution 2                Solution 3
A    50%    1             A    50%    0             A    50%    0
B    25%    0             B    25%    1             B    25%    0
C    60%    0             C    60%    0             C    60%    0
D    50%    1             D    50%    0             D    50%    0


Therefore, there are three distinct possible definitions of "correct" in the second problem, leading to either A or D being correct (in solution 1) or only B being correct (in solution 2), or none of the answers being correct (in solution 3).

One way to interpret this state of affairs is that for each of the answers A, B, and D, there exists at least one way of marking the tickets that makes those answers correct. This does not imply that all three are simultaneously correct: they couldn't be, because $.25 \ne .50$. If you were the grader of the test, then if you marked any of A, B, or D correct, then you would not get an argument from the test-taker; but if you marked any of them incorrect, the test-taker would have a legitimate basis to dispute your scoring: they would invoke either solution 1 or solution 2. Indeed, if a test-taker refused to answer the question, solution 3 would give them a legitimate basis to argue that their non-response ought to get full credit, too!

In summary, this analysis addresses the second part of the question by concluding that any of the following responses to question 2 should be marked correct because each of them are defensible: A, B, D, A and D, and nothing. No other response can be defended and therefore would not be correct.

-
So what is your answer to question 2? It seems like you have given an immensely complicated explanation with three possible consistent answers. I still maintain that this indicates a problem with the definition of correct. –  Michael Chernick Jun 13 '12 at 2:33
@Michael Thank you: I have added a paragraph to make the conclusions perfectly clear. I will admit to crafting a relatively long reply (which perhaps is justified by the much greater collective length of replies here and on the math site). "Immensely complicated" must be in the eye of the beholder: I have worked to make the ideas as straightforward as possible so that readers can easily check that I'm not trying to deceive them. When others claim there is a "paradox" or a "question in logic," it essential to be simple, clear, and explicit, although doing so may increase the length. –  whuber Jun 13 '12 at 14:07
I know that you write well and are usually very clear in what you say. I find this response complicated because I don't fully understand it. I don't see that what you say eliminates the paradox and the circular argument. –  Michael Chernick Jun 13 '12 at 14:45
@Michael I appreciate that. I'm a little diffident about this resolution myself, because people are not comfortable with the idea that there can be multiple correct answers to clear simple questions. I invite critical evaluation of this idea of "defensibility" as a way around the apparent paradox. To me, it seems like an original way to circumvent the negative conclusions reached by others (namely, that the second question is meaningless, or nonsense, or illogical). The main value of examining paradoxes lies in encouraging deeper examination of fundamental ideas. –  whuber Jun 13 '12 at 14:50
You gave three answers that you say are defensible. How do you defend each answer? –  Michael Chernick Jun 13 '12 at 14:58
show 5 more comments

I think there is an issue of semantics here in addition to probability. Choosing at random is clear. Each of A, B, C, and D will be selected 25%. But what does it mean to be correct when you pick at random? It seems that it should mean given that you pick A does As answer give the correct % of samples that will be correct and the same for B, C, and D. So you have to count 1/4 for each correct answer and sum over all correct answers to get the correct percentage. But this leads to a circular argument. Hence the paradox. This really seems to be a question in logic rather than probability or statistics.

-
+1. I was asking myself "But what does it mean to be correct...?". I agree this seems to be more of a logic puzzle than a probability question (although I'd like to hear an explanation from someone about why this perception is wrong). –  Macro Jun 12 '12 at 18:14
Logic behind the original question was just as you have described. As an answer 25% has 50% probability to be choosen, and an answers 50% and 60% have 25% probability to be choosen, this answers are not correct. The answer 0% is correct, as probability to choose it is 0%. This reminds circular argument, but does this make the question incorrect? –  Nick Jun 12 '12 at 18:31
@Nick I don't think so. I think the circular argument makes it indeterminate. You can't say which answers are correct and you can't say which answers are incorrect. So 0% is not the answer. The question cannot be answered. Perhaps you can say that 60% is incorrect because if there were an answer it would have to be a multiple of 1/4. –  Michael Chernick Jun 12 '12 at 18:44