# What will be the correct answer, if we modify the “Best statistics question ever”?

There is a popular question, called "Best statistics question ever".

If you choose an answer to this question at random, what is the chance you will be correct?

A) 25% B) 50% C) 60% D) 25%

This task is not very difficult, the correct answer is 0%. But if we modify it like this:

If you choose an answer to this question at random, what is the chance you will be correct?

A) 50% B) 25% C) 60% D) 50%

What will be the correct answer? Do we have two correct answers: 25% and 50%, or there is no correct answer, as with this two correct answers the chance of choosing the correct answer is in fact 75% (but we do not have 75% written on the desk)?

By the way. Does the answer 0% remains correct answer, the third correct answer in this case?

• The answer, of course, depends on how the random choice is made. "At random" does not always mean "uniformly at random"... ;) – MånsT Jun 12 '12 at 18:10
• Lets assume "uniformely at random". Logic behind the original question was:As an answer 25% has 50% probability to be choosen, and an answers 50% and 60% have 25% probability to be choosen, this answers are not correct. The answer 0% is correct, as probability to choose it is 0%. – Nick Jun 12 '12 at 18:32
• This problem was exhaustively analyzed on math.SE awhile back. – cardinal Jun 12 '12 at 18:32
• @whuber: Here's the one with the most votes. There were others, but maybe they got closed/merged. – cardinal Jun 12 '12 at 19:08
• @Cardinal Thanks. I guess I'm glad to see there appear to be no deep or illuminating replies there (despite the heavy voting), because it means there may be some scope for further exegesis here :-). – whuber Jun 12 '12 at 19:12

The apparent paradoxes (of logic or probability) can be resolved by framing the questions clearly and carefully.

The following analysis is motivated by the idea of defending an answer: when a test-taker can exhibit a possible state of affairs (consistent with all available information) in which their answer indeed is correct, then it should be marked as correct. Equivalently, an answer is incorrect when no such defense exists; it is considered correct otherwise. This models the usual interactions between (benevolent, rational) graders and (rational) test-takers :-). The apparent paradox is resolved by exhibiting multiple such defenses for the second question, only one of which could apply in any instance.

I will take the meaning of "random" in these questions in a conventional sense: to model a random choice of answer, I will write each answer on a slip of paper ("ticket") and put it in a box: that will be four tickets total. Drawing a ticket out of the box (after carefully and blindly shuffling the box's contents) is a physical model for a "random" choice. It motivates and justifies a corresponding probability model.

Now, what does it mean to "be correct"? In my ignorance, I will explore all possibilities. In any case, I take it as definite that zero, one, or even more of the tickets may be "correct." (How might I know? I simply consult the grading sheet!) I will mark the "correct" answers as such by writing the value $1$ on each correct ticket and writing $0$ on the others. That's routine and should not be controversial.

An obvious but important thing to notice is that the rule for writing $0$ or $1$ must be based solely on the answer written on each ticket: mathematically, it is a mapping (or reassignment) sending the set of listed answers ($\{.25, .50, .60\}$ in both questions) into the set $\{0,1\}$. This rule is needed for self-consistency.

Let's turn to the probabilistic element of the question: by definition, the chance of being correct, under a random drawing of tickets, is the expectation of the values with which they have been marked. The expectation is computed by summing the values on the tickets and dividing that by their total number. It will therefore be either $0$, $.25$, $.50$, $.75$, or $1$.

A marking will make sense provided that only the tickets whose answers equal the expectation are marked with $1$s. This also is a self-consistency requirement. I claim that this is the crux of the matter: to find and interpret the markings that make sense. If there are none, then the question itself can be branded as being meaningless. It there is a unique marking, then there will be no controversy. Only if two or more markings make sense will there be any potential difficulty.

Which markings make sense?

We don't even need to make an exhaustive search. In the first question, the expectations listed on the tickets are 25%, 50% and 60%. The latter is impossible with four tickets. The first would require exactly one ticket to be marked; the second, two tickets. That gives at most $3+3=6$ possible markings to explore. The only marking that makes sense puts $0$s on each ticket. For this marking, the expectation is $(0+0+0+0)/4 = 0$. That justifies the stated answer to the first question. (Arguably, the sole correct response to the first question is not to select any answer!)

In the second question, the same answers appear and once again there are six markings to explore. This time, three markings are self-consistent. I tabulate them:

Solution 1                Solution 2                Solution 3
A    50%    1             A    50%    0             A    50%    0
B    25%    0             B    25%    1             B    25%    0
C    60%    0             C    60%    0             C    60%    0
D    50%    1             D    50%    0             D    50%    0


Therefore, there are three distinct possible definitions of "correct" in the second problem, leading to either A or D being correct (in solution 1) or only B being correct (in solution 2), or none of the answers being correct (in solution 3).

One way to interpret this state of affairs is that for each of the answers A, B, and D, there exists at least one way of marking the tickets that makes those answers correct. This does not imply that all three are simultaneously correct: they couldn't be, because $.25 \ne .50$. If you were the grader of the test, then if you marked any of A, B, or D correct, then you would not get an argument from the test-taker; but if you marked any of them incorrect, the test-taker would have a legitimate basis to dispute your scoring: they would invoke either solution 1 or solution 2. Indeed, if a test-taker refused to answer the question, solution 3 would give them a legitimate basis to argue that their non-response ought to get full credit, too!

In summary, this analysis addresses the second part of the question by concluding that any of the following responses to question 2 should be marked correct because each of them are defensible: A, B, D, A and D, and nothing. No other response can be defended and therefore would not be correct.

• So what is your answer to question 2? It seems like you have given an immensely complicated explanation with three possible consistent answers. I still maintain that this indicates a problem with the definition of correct. – Michael R. Chernick Jun 13 '12 at 2:33
• @Michael Thank you: I have added a paragraph to make the conclusions perfectly clear. I will admit to crafting a relatively long reply (which perhaps is justified by the much greater collective length of replies here and on the math site). "Immensely complicated" must be in the eye of the beholder: I have worked to make the ideas as straightforward as possible so that readers can easily check that I'm not trying to deceive them. When others claim there is a "paradox" or a "question in logic," it essential to be simple, clear, and explicit, although doing so may increase the length. – whuber Jun 13 '12 at 14:07
• @Michael I appreciate that. I'm a little diffident about this resolution myself, because people are not comfortable with the idea that there can be multiple correct answers to clear simple questions. I invite critical evaluation of this idea of "defensibility" as a way around the apparent paradox. To me, it seems like an original way to circumvent the negative conclusions reached by others (namely, that the second question is meaningless, or nonsense, or illogical). The main value of examining paradoxes lies in encouraging deeper examination of fundamental ideas. – whuber Jun 13 '12 at 14:50
• Ah, that's the point, @Michael: even though the three are contradictory, each is defensible. I provided the defense in the explanation leading up to the tabulated solutions. Let's get practical here: imagine you are a teacher at an Ivy League institution (where students and their parents, unfortunately, have become rather litigious in recent years) and you have given such questions as part of an exam. How do you grade them? My analysis suggests a way that is rational, objective, fair, and avoids argument between teacher and student--even though the students may differ among themselves! – whuber Jun 13 '12 at 15:04
• There's your problem, @Michael: the assumption that there must be a unique response to a multiple choice question. (Talk to the physicians you know and ask them about the multiple choice questions on their board certification exams: at least in the past, several responses to each question would be correct.) Here, the presence of two identical responses to the second question indicates the possibility that more than one response to a question could be considered correct.// At this point we seem in danger of veering away from the issue, so if you have further remarks, please start a chat room. – whuber Jun 13 '12 at 18:31

I think there is an issue of semantics here in addition to probability. Choosing at random is clear. Each of A, B, C, and D will be selected 25%. But what does it mean to be correct when you pick at random? It seems that it should mean given that you pick A does As answer give the correct % of samples that will be correct and the same for B, C, and D. So you have to count 1/4 for each correct answer and sum over all correct answers to get the correct percentage. But this leads to a circular argument. Hence the paradox. This really seems to be a question in logic rather than probability or statistics.

• +1. I was asking myself "But what does it mean to be correct...?". I agree this seems to be more of a logic puzzle than a probability question (although I'd like to hear an explanation from someone about why this perception is wrong). – Macro Jun 12 '12 at 18:14
• Logic behind the original question was just as you have described. As an answer 25% has 50% probability to be choosen, and an answers 50% and 60% have 25% probability to be choosen, this answers are not correct. The answer 0% is correct, as probability to choose it is 0%. This reminds circular argument, but does this make the question incorrect? – Nick Jun 12 '12 at 18:31
• @Nick I don't think so. I think the circular argument makes it indeterminate. You can't say which answers are correct and you can't say which answers are incorrect. So 0% is not the answer. The question cannot be answered. Perhaps you can say that 60% is incorrect because if there were an answer it would have to be a multiple of 1/4. – Michael R. Chernick Jun 12 '12 at 18:44

whuber gives a great analysis where multiple answers are allowed. However, there is also a consistent way to understand the question such that there is only a single correct answer (though we need to state this as part of the question):

If you choose an answer to this question at random, what is the chance you will be correct, given that there is only a single correct answer?

A) 50% B) 25% C) 60% D) 50%

Again, we will define a "correct" answer as one that is rationally defensible and will follow the argument of whuber. A marking is a map from the set of answers to $\{0,1\}$ such that correct answers are sent to 1 and incorrect answers to 0. There are three possible self-consistent markings:

Solution 1                Solution 2                Solution 3
A    50%    1             A    50%    0             A    50%    0
B    25%    0             B    25%    1             B    25%    0
C    60%    0             C    60%    0             C    60%    0
D    50%    1             D    50%    0             D    50%    0


Yet we need another logical step to narrow these down to a single defensible answer. When making this problem, the teacher was faced with three possible markings, each of which could be as equally an acceptable single answer as the others. However, as only one answer can be correct, the teacher should choose randomly between them. This assigns an equal probability to each marking so that:

• $1/3$ of the time the student will be right $50\%$ of the time
• $1/3$ of the time the student will be right $25\%$ of the time
• $1/3$ of the time the student will be right $0\%$ of the time

This results in the expectation that the student will be right $(50+25+0)/3=25$% of the time. Thus 25% should be the correct answer and the marking for Solution 2 should be chosen. This is a self-consistent update of the teacher's prior over the three possible markings, i.e. if Solution 2 is fixed to be the correct marking 100% of the time, then 25% is still the one correct answer.

Summary: If we specify that there is only a single correct answer, then that answer is 25%.

• Could you explain the justification for "However, as only one answer can be correct, the teacher should choose randomly between them"? I do not see why a random choice is required at this juncture. – whuber Sep 30 '15 at 0:08
• Of course it is possible that the teacher could have some kind of bias, but there is no reason to think that there is one. In the complete absence of any other information, I think the rational prior to assume is that the choice follows a uniform distribution. Let me know if this seems mistaken. – W. J. Zeng Oct 1 '15 at 2:28
• I don't know whether it is mistaken or not, but calling it "the rational prior" doesn't seem like any explanation at all. What bothers me is that to "specify that there is only a single correct answer" could be inherently contradictory, in which case no prior distribution will be relevant. Moreover, the logical outline of your argument seems to be "if we assume there is a single correct answer and we also adopt a prior distribution among the three possible correct answers, then we conclude there is a unique correct answer." That looks awfully circular, but maybe it could be fixed up. – whuber Oct 1 '15 at 3:17

I believe the answer is 1/3. We don't know which answer (25%, 50%, or 60%) is correct. So, each answer, 25%, 50%, and 60% has a 1/3 chance of being correct if selected. Even though 25% appears twice, it still has a 1/3 chance of being the correct answer. It actually does not matter how many times 25% appears as an answer. If it appears 10 times along with the 50% and the 60%, the chance that it is the correct answer would still be 1/3.This assumes that one of the answers is correct. If there is a possibility of none of the answers being correct, then the answer would be 1/4. This is based on my interpretation of what the question is asking.

• This attempt at an explanation fails because it invalidly conflates ignorance (we don't know which of three answers would be correct) with probability (assigning 1/3 to each answer). – whuber Jan 7 '14 at 21:43