# Did Deborah Mayo refute Birnbaum's proof of the likelihood principle?

This is somewhat related to my previous question here: An example where the likelihood principle *really* matters?

Apparently, Deborah Mayo published a paper in Statistical Science refuting Birnbaum's proof of the likelihood principle. Can anyone explain the main argument by Birnbaum and the counter-argument by Mayo? Is she right (logically)?

• A refutation of this refutation: academic.oup.com/bjps/article-abstract/66/3/475/1497890 (free copy: gandenberger.org/wp-content/uploads/2013/11/…). Dec 4, 2018 at 20:59
• More commentary: stats.stackexchange.com/questions/65520/… Dec 5, 2018 at 16:05
• The problem with this is that Birnbaum's original proof is not mathematically precise, as it does not define what an "inference" is, as a mathematical object. It can be made precise and this has happened later, but there is more than one way of doing it. As a consequence of this, there are versions that can be refuted by Mayo's argument and versions that cannot be refuted. In my interpretation Mayo's argument states that the weak conditionality principle and the sufficiency principle as required in the proof cannot both hold at the same time. (To be continued.) Jan 9, 2020 at 14:05
• This would mean that the proof in itself is correct but its assumptions can never be fulfilled. There is another version of the proof that uses a different formalisation of the two principles so that they can be fulfilled at the same time, and the proof is still valid. But this makes the principles so strong that they basically directly demand that p-values (and anything that violates the likelihood principle) do not count as an "inference", in which case the proof doesn't show much of interest, and surely is no reason to abandon frequentist methods, as these were excluded by assumption. Jan 9, 2020 at 14:10

In a nutshell, Birnbaum's argument is that two widely accepted principles logically imply that the likelihood principle must hold. The counter-argument of Mayo is that the proof is wrong because Birnbaum misuses one of the principles.

Below I simplify the arguments to the extent that they are not very rigorous. My purpose is to make them accessible to a wider audience because the original arguments are very technical. Interested readers should see the detail in the articles linked in the question and in the comments.

For the sake of concreteness, I will focus on the case of a coin with unknown bias $$\theta$$. In experiment $$E_1$$ we flip it 10 times. In experiment $$E_2$$ we flip it until we obtain 3 "tails". In experiment $$E_{mix}$$ we flip a fair coin with labels "1" and "2" on either side: if it lands a "1" we perform $$E_1$$; if it lands a "2" we perform $$E_2$$. This example will greatly simplify the discussion and will exhibit the logic of the arguments (the original proofs are of course more general).

The principles:

The following two principles are widely accepted:

The Weak Conditionality Principle says that we should draw the same conclusions if we decide to perform experiment $$E_1$$, or if we decide to perform $$E_{mix}$$ and the coin lands "1".

The Sufficiency Principle says that we should draw the same conclusions in two experiments where a sufficient statistic has the same value.

The following principle is accepted by the Bayesian but not by the frequentists. Yet, Birnbaum claims that it is a logical consequence of the first two.

The Likelihood Principle says that we should draw the same conclusions in two experiments where the likelihood functions are proportional.

Birnbaum's theorem:

Say we perform $$E_1$$ and we obtain 7 "heads" out of ten flips. The likelihood function of $$\theta$$ is $${10 \choose 3}\theta^7(1-\theta)^3$$. We perform $$E_2$$ and need to flip the coin 10 times to obtain 3 "tails". The likelihood function of $$\theta$$ is $${9 \choose 7}\theta^7(1-\theta)^3$$. The two likelihood functions are proportional.

Birnbaum considers the following statistic on $$E_{mix}$$ from $$\{1, 2\} \times \mathbb{N}^2$$ to $$\{1, 2\} \times \mathbb{N}^2$$: $$T: (\xi, x,y) \rightarrow (1, x,y),$$ where $$x$$ and $$y$$ are the numbers of "heads" and "tails", respectively. So no matter what happens, $$T$$ reports the result as if it came from experiment $$E_1$$. It turns out that $$T$$ is sufficient for $$\theta$$ in $$E_{mix}$$. The only case that is non-trivial is when $$x = 7$$ and $$y = 3$$, where we have

$$P(X_{mix}=(1,x,y)|T=(1,x,y)) = \frac{0.5 \times {10 \choose 3}\theta^7(1-\theta)^3}{0.5 \times {10 \choose 3}\theta^7(1-\theta)^3 + 0.5 \times {9 \choose 7}\theta^7(1-\theta)^3}\\=\frac{{10 \choose 3}}{{10 \choose 3}+{9 \choose 7}}\text{, a value that is independent of } \theta.$$ All the other cases are 0 or 1—except $$P(X_{mix}=(2,x,y)|T=(1,x,y))$$, which is the complement of the probability above. The distribution of $$X_{mix}$$ given $$T$$ is independent of $$\theta$$, so $$T$$ is a sufficient statistic for $$\theta$$.

Now, according to the sufficiency principle, we must conclude the same for $$(1,x,y)$$ and $$(2,x,y)$$ in $$E_{mix}$$, and from the weak condionality principle, we must conclude the same for $$(x,y)$$ in $$E_1$$ and $$(1,x,y)$$ in $$E_{mix}$$, as well as for $$(x,y)$$ in $$E_2$$ and $$(2,x,y)$$ in $$E_{mix}$$. So our conclusion must be the same in all cases, which is the likelihood principle.

Mayo's counter-proof:

The setup of Birnbaum is not a mixture experiment because the result of the coin labelled "1" and "2" was not observed, therefore the weak conditionality principle does not apply to this case.

Take the test $$\theta = 0.5$$ versus $$\theta > 0.5$$ and draw a conclusion from the p-value of the test. As a preliminary observation, note that the p-value of $$(7,3)$$ in $$E_1$$ is given by the binomial distribution as approximately $$0.1719$$; the p-value of $$(7,3)$$ in $$E_2$$ is given by the negative binomial distribution as approximately $$0.0898$$.

Here comes the important part: the p-value of $$T=(1,7,3)$$ in $$E_{mix}$$ is given as the average of the two—remember we do not know the status of the coin—i.e. approximately $$0.1309$$. Yet the p-value of $$(1,7,3)$$ in $$E_{mix}$$—where the coin is observed—is the same as that in $$E_1$$, i.e. approximately $$0.1719$$. The weak conditionality principle holds (the conclusion is the same in $$E_1$$ and in $$E_{mix}$$ where the coin lands "1") and yet the likelihood principle does not. The counter-example disproves Birnbaum's theorem.

Peña and Berger's refutation of Mayo's counter-proof:

Mayo implicitly changed the statement of the sufficiency principle: she interprets "same conclusions" as "same method". Taking the p-value is an inference method, but not a conclusion. This is important because an agent can come to identical conclusions even when two p-values are different. This is not meant in the sense that you accept the null hypothesis if the p-value is 0.8 or 0.9, but in the sense that the two p-values of Mayo are computed from different experiments (different probability spaces with different outcomes), so with this information at hand you can draw the same conclusion even if the values are different.

The sufficiency principle says that if there exists a sufficient statistic, then the conclusions must be the same, but it does not require the sufficient statistic to be used at all. If it did, it would lead to a contradiction, as demonstrated by Mayo.

• As a side note, one may question the value of founding principles if nobody can really tell when and how they apply. I wonder why the axiomatic method works well for probability but not so much for the theory of statistics. Apr 22, 2019 at 9:14
• The discussion relies crucially on how the weak conditionality principle and the sufficiency principle are exactly formulated and applied in the proof, which isn't quite clear in the posting. I don't want to criticise the poster, as it's very difficult to elaborate these things completely, and what is written isn't wrong. But logically it can't be clear to the reader from this whether Mayo is right, or whether Pena and Berger are right - and actually it depends on the precise formulation and there are possibilities to do this that would make right the claim of either side. Jan 9, 2020 at 14:19
• @Lewian thanks for your comment. I agree with you, the detail matters, which makes it hard to simplify the ideas (someone with the skills could just look at the original articles). From what I gathered, it's not clear to anyone who is right: those who hate the likelihood principle seem to agree with Mayo, those who like it seem to disagree with her. Anyway, if you have ideas of how to improve the post, please go ahead, I won't be offended. Quite the reverse, I wish to make it as useful as possible, but I am far from an expert on this topic. Jan 10, 2020 at 16:08
• Actually I believe I have a pretty good understanding of this, and I have discussed it with various people including D. Mayo herself and (from the other side) Phil Dawid, although both of them would think the other one is wrong and wouldn't therefore agree with my defense of what the other one is saying. Unfortunately it is complicated enough that I don't think it's realistic that I can find the time to properly elaborate it for posting it here any time soon, which of course means that nobody needs to believe what I claim... Jan 10, 2020 at 16:13
• @Lewian that's amazing! Would you at least have the time to briefly sketch the arguments and point to existing references? If so, you could send it to me (easy to find the contact details from my profile) and perhaps I could improve the answer. I cannot do this soon either, but one day I could come back to it... Jan 10, 2020 at 17:07

While it is interesting to determine the validity of Birnbaum’s (1962) proof that the sufficiency principle (SP) and one of the versions of the conditionality principle (CP) together imply the likelihood principle (LP), I believe there is a deeper problem with the theorem. Specifically, the CP cannot be justified from a conditional inference perspective. The reasoning is as follows.

Fisher provided a number of examples showing that conditioning on an ancillary statistic was a reasonable approach. The CP is the result of extrapolating these small number of examples into a principle that requires us to always condition on an ancillary when one exists. The question is: Is this a case of unjustified extrapolation? I believe so. Consider first the intuitive argument for the mixture example. Informally, (from Giere 1977), the weak conditionality principle (WCP) claims the “irrelevance of (component) experiments not actually performed” while the LP claims “the irrelevance of outcomes not actually observed”. It is easy to see the intuitive appeal of the WCP when described this way. However, WCP could also be described informally as claiming “the irrelevance of some outcomes not actually observed.” Frequentists are concerned to use an appropriate reference set, but how do we know that conditioning on this ancillary variable provides the best reference set to use? The answer is we do not.

To see this, imagine a mixture model with two ancillaries, A and B, where A is the flip of the coin in the mixture experiment and where conditioning on A leads to much weaker inferences than conditioning on B. Ancillary B reflects a partition of the sample space across both components experiments. Thus, conditioning on B is preferred over A in the mixture model but is not available to use for conditioning in either component experiment. Not only will the WCP fail for this admittedly special case, it heralds a more-common problem. Perturb this first mixture model slightly so that A is still ancillary but B is now only approximately-ancillary. Conditioning on B will still be preferred over A even though A is the unique ancillary in the modified problem. In short, the conditionality principles cannot be justified because there can be better statistics to condition on than ancillaries. [Cox’s comment on Buehler’s (1982) ancillary paper discusses the need for forms of inference to be robust to minor perturbations of model specification. The WCP fails this test.]

Finally, as an aside, some history that tends to be overlooked in discussions about Birnbaum’s theorem. Giere (1977) reports that Birnbaum rejected the Likelihood Principle within two years of the publication of his theorem. Birnbaum abandoned the LP in favor of what he called the confidence concept in his 1969 paper.