This is somewhat related to my previous question here: An example where the likelihood principle *really* matters?

Apparently, Deborah Mayo published a paper in Statistical Science refuting Birnbaum's proof of the likelihood principle. Can anyone explain the main argument by Birnbaum and the counter-argument by Mayo? Is she right (logically)?


In a nutshell, Birnbaum's argument is that two widely accepted principles logically imply that the likelihood principle must hold. The counter-argument of Mayo is that the proof is wrong because Birnbaum misuses one of the principles.

Below I simplify the arguments to the extent that they are not very rigorous. My purpose is to make them accessible to a wider audience because the original arguments are very technical. Interested readers should see the detail in the articles linked in the question and in the comments.

For the sake of concreteness, I will focus on the case of a coin with unknown bias $\theta$. In experiment $E_1$ we flip it 10 times. In experiment $E_2$ we flip it until we obtain 3 "tails". In experiment $E_{mix}$ we flip a fair coin with labels "1" and "2" on either side: if it lands a "1" we perform $E_1$; if it lands a "2" we perform $E_2$. This example will greatly simplify the discussion and will exhibit the logic of the arguments (the original proofs are of course more general).

The principles:

The following two principles are widely accepted:

The Weak Conditionality Principle says that we should draw the same conclusions if we decide to perform experiment $E_1$, or if we decide to perform $E_{mix}$ and the coin lands "1".

The Sufficiency Principle says that we should draw the same conclusions in two experiments where a sufficient statistic has the same value.

The following principle is accepted by the Bayesian but not by the frequentists. Yet, Birnbaum claims that it is a logical consequence of the first two.

The Likelihood Principle says that we should draw the same conclusions in two experiments where the likelihood functions are proportional.

Birnbaum's theorem:

Say we perform $E_1$ and we obtain 7 "heads" out of ten flips. The likelihood function of $\theta$ is ${10 \choose 3}\theta^7(1-\theta)^3$. We perform $E_2$ and need to flip the coin 10 times to obtain 3 "tails". The likelihood function of $\theta$ is ${9 \choose 7}\theta^7(1-\theta)^3$. The two likelihood functions are proportional.

Birnbaum considers the following statistic on $E_{mix}$ from $\{1, 2\} \times \mathbb{N}^2$ to $\{1, 2\} \times \mathbb{N}^2$: $$T: (\xi, x,y) \rightarrow (1, x,y),$$ where $x$ and $y$ are the numbers of "heads" and "tails", respectively. So no matter what happens, $T$ reports the result as if it came from experiment $E_1$. It turns out that $T$ is sufficient for $\theta$ in $E_{mix}$. The only case that is non-trivial is when $x = 7$ and $y = 3$, where we have

$$P(X_{mix}=(1,x,y)|T=(1,x,y)) = \frac{0.5 \times {10 \choose 3}\theta^7(1-\theta)^3}{0.5 \times {10 \choose 3}\theta^7(1-\theta)^3 + 0.5 \times {9 \choose 7}\theta^7(1-\theta)^3}\\=\frac{{10 \choose 3}}{{10 \choose 3}+{9 \choose 7}}\text{, a value that is independent of } \theta.$$ All the other cases are 0 or 1—except $P(X_{mix}=(2,x,y)|T=(1,x,y))$, which is the complement of the probability above. The distribution of $X_{mix}$ given $T$ is independent of $\theta$, so $T$ is a sufficient statistic for $\theta$.

Now, according to the sufficiency principle, we must conclude the same for $(1,x,y)$ and $(2,x,y)$ in $E_{mix}$, and from the weak condionality principle, we must conclude the same for $(x,y)$ in $E_1$ and $(1,x,y)$ in $E_{mix}$, as well as for $(x,y)$ in $E_2$ and $(2,x,y)$ in $E_{mix}$. So our conclusion must be the same in all cases, which is the likelihood principle.

Mayo's counter-proof:

The setup of Birnbaum is not a mixture experiment because the result of the coin labelled "1" and "2" was not observed, therefore the weak conditionality principle does not apply to this case.

Take the test $\theta = 0.5$ versus $\theta > 0.5$ and draw a conclusion from the p-value of the test. As a preliminary observation, note that the p-value of $(7,3)$ in $E_1$ is given by the binomial distribution as approximately $0.1719$; the p-value of $(7,3)$ in $E_2$ is given by the negative binomial distribution as approximately $0.0898$.

Here comes the important part: the p-value of $T=(1,7,3)$ in $E_{mix}$ is given as the average of the two—remember we do not know the status of the coin—i.e. approximately $0.1309$. Yet the p-value of $(1,7,3)$ in $E_{mix}$—where the coin is observed—is the same as that in $E_1$, i.e. approximately $0.1719$. The weak conditionality principle holds (the conclusion is the same in $E_1$ and in $E_{mix}$ where the coin lands "1") and yet the likelihood principle does not. The counter-example disproves Birnbaum's theorem.

Peña and Berger's refutation of Mayo's counter-proof:

Mayo implicitly changed the statement of the sufficiency principle: she interprets "same conclusions" as "same method". Taking the p-value is an inference method, but not a conclusion. This is important because an agent can come to identical conclusions even when two p-values are different. This is not meant in the sense that you accept the null hypothesis if the p-value is 0.8 or 0.9, but in the sense that the two p-values of Mayo are computed from different experiments (different probability spaces with different outcomes), so with this information at hand you can draw the same conclusion even if the values are different.

The sufficiency principle says that if there exists a sufficient statistic, then the conclusions must be the same, but it does not require the sufficient statistic to be used at all. If it did, it would lead to a contradiction, as demonstrated by Mayo.

  • 2
    $\begingroup$ As a side note, one may question the value of founding principles if nobody can really tell when and how they apply. I wonder why the axiomatic method works well for probability but not so much for the theory of statistics. $\endgroup$ – gui11aume Apr 22 '19 at 9:14
  • $\begingroup$ The discussion relies crucially on how the weak conditionality principle and the sufficiency principle are exactly formulated and applied in the proof, which isn't quite clear in the posting. I don't want to criticise the poster, as it's very difficult to elaborate these things completely, and what is written isn't wrong. But logically it can't be clear to the reader from this whether Mayo is right, or whether Pena and Berger are right - and actually it depends on the precise formulation and there are possibilities to do this that would make right the claim of either side. $\endgroup$ – Lewian Jan 9 at 14:19
  • $\begingroup$ @Lewian thanks for your comment. I agree with you, the detail matters, which makes it hard to simplify the ideas (someone with the skills could just look at the original articles). From what I gathered, it's not clear to anyone who is right: those who hate the likelihood principle seem to agree with Mayo, those who like it seem to disagree with her. Anyway, if you have ideas of how to improve the post, please go ahead, I won't be offended. Quite the reverse, I wish to make it as useful as possible, but I am far from an expert on this topic. $\endgroup$ – gui11aume Jan 10 at 16:08
  • $\begingroup$ Actually I believe I have a pretty good understanding of this, and I have discussed it with various people including D. Mayo herself and (from the other side) Phil Dawid, although both of them would think the other one is wrong and wouldn't therefore agree with my defense of what the other one is saying. Unfortunately it is complicated enough that I don't think it's realistic that I can find the time to properly elaborate it for posting it here any time soon, which of course means that nobody needs to believe what I claim... $\endgroup$ – Lewian Jan 10 at 16:13
  • $\begingroup$ @Lewian that's amazing! Would you at least have the time to briefly sketch the arguments and point to existing references? If so, you could send it to me (easy to find the contact details from my profile) and perhaps I could improve the answer. I cannot do this soon either, but one day I could come back to it... $\endgroup$ – gui11aume Jan 10 at 17:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.