This is somewhat related to my previous question here: An example where the likelihood principle *really* matters?
Apparently, Deborah Mayo published a paper in Statistical Science refuting Birnbaum's proof of the likelihood principle. Can anyone explain the main argument by Birnbaum and the counter-argument by Mayo? Is she right (logically)?
In a nutshell, Birnbaum's argument is that two widely accepted principles logically imply that the likelihood principle must hold. The counter-argument of Mayo is that the proof is wrong because Birnbaum misuses one of the principles.
Below I simplify the arguments to the extent that they are not very rigorous. My purpose is to make them accessible to a wider audience because the original arguments are very technical. Interested readers should see the detail in the articles linked in the question and in the comments.
For the sake of concreteness, I will focus on the case of a coin with unknown bias $\theta$. In experiment $E_1$ we flip it 10 times. In experiment $E_2$ we flip it until we obtain 3 "tails". In experiment $E_{mix}$ we flip a fair coin with labels "1" and "2" on either side: if it lands a "1" we perform $E_1$; if it lands a "2" we perform $E_2$. This example will greatly simplify the discussion and will exhibit the logic of the arguments (the original proofs are of course more general).
The principles:
The following two principles are widely accepted:
The Weak Conditionality Principle says that we should draw the same conclusions if we decide to perform experiment $E_1$, or if we decide to perform $E_{mix}$ and the coin lands "1".
The Sufficiency Principle says that we should draw the same conclusions in two experiments where a sufficient statistic has the same value.
The following principle is accepted by the Bayesian but not by the frequentists. Yet, Birnbaum claims that it is a logical consequence of the first two.
The Likelihood Principle says that we should draw the same conclusions in two experiments where the likelihood functions are proportional.
Birnbaum's theorem:
Say we perform $E_1$ and we obtain 7 "heads" out of ten flips. The likelihood function of $\theta$ is ${10 \choose 3}\theta^7(1-\theta)^3$. We perform $E_2$ and need to flip the coin 10 times to obtain 3 "tails". The likelihood function of $\theta$ is ${9 \choose 7}\theta^7(1-\theta)^3$. The two likelihood functions are proportional.
Birnbaum considers the following statistic on $E_{mix}$ from $\{1, 2\} \times \mathbb{N}^2$ to $\{1, 2\} \times \mathbb{N}^2$: $$T: (\xi, x,y) \rightarrow (1, x,y),$$ where $x$ and $y$ are the numbers of "heads" and "tails", respectively, and $\xi$ is the outcome of the fair coin with labels "1" and "2". So no matter what happens, $T$ reports the result as if it came from experiment $E_1$. It turns out that $T$ is sufficient for $\theta$ in $E_{mix}$. The only case that is non-trivial is when $x = 7$ and $y = 3$, where we have
$$P(X_{mix}=(1,x,y)|T=(1,x,y)) = \frac{0.5 \times {10 \choose 3}\theta^7(1-\theta)^3}{0.5 \times {10 \choose 3}\theta^7(1-\theta)^3 + 0.5 \times {9 \choose 7}\theta^7(1-\theta)^3}\\=\frac{{10 \choose 3}}{{10 \choose 3}+{9 \choose 7}}\text{, a value that is independent of } \theta.$$ All the other cases are 0—except $P(X_{mix}=(2,x,y)|T=(1,x,y))$, which is the complement of the probability above. The distribution of $X_{mix}$ given $T$ is independent of $\theta$, so $T$ is a sufficient statistic for $\theta$.
Now, according to the sufficiency principle, we must conclude the same for $(1,x,y)$ and $(2,x,y)$ in $E_{mix}$, and from the weak condionality principle, we must conclude the same for $(x,y)$ in $E_1$ and $(1,x,y)$ in $E_{mix}$, as well as for $(x,y)$ in $E_2$ and $(2,x,y)$ in $E_{mix}$. So our conclusion must be the same in all cases, which is the likelihood principle.
Mayo's counter-proof:
The setup of Birnbaum is not a mixture experiment because the result of the coin labelled "1" and "2" was not observed, therefore the weak conditionality principle does not apply to this case.
Take the test $\theta = 0.5$ versus $\theta > 0.5$ and draw a conclusion from the p-value of the test. As a preliminary observation, note that the p-value of $(7,3)$ in $E_1$ is given by the binomial distribution as approximately $0.1719$; the p-value of $(7,3)$ in $E_2$ is given by the negative binomial distribution as approximately $0.0898$.
Here comes the important part: the p-value of $T=(1,7,3)$ in $E_{mix}$ is given as the average of the two—remember we do not know the status of the coin—i.e. approximately $0.1309$. Yet the p-value of $(1,7,3)$ in $E_{mix}$—where the coin is observed—is the same as that in $E_1$, i.e. approximately $0.1719$. The weak conditionality principle holds (the conclusion is the same in $E_1$ and in $E_{mix}$ where the coin lands "1") and yet the likelihood principle does not. The counter-example disproves Birnbaum's theorem.
Peña and Berger's refutation of Mayo's counter-proof:
Mayo implicitly changed the statement of the sufficiency principle: she interprets "same conclusions" as "same method". Taking the p-value is an inference method, but not a conclusion. This is important because an agent can come to identical conclusions even when two p-values are different. This is not meant in the sense that you accept the null hypothesis if the p-value is 0.8 or 0.9, but in the sense that the two p-values of Mayo are computed from different experiments (different probability spaces with different outcomes), so with this information at hand you can draw the same conclusion even if the values are different.
The sufficiency principle says that if there exists a sufficient statistic, then the conclusions must be the same, but it does not require the sufficient statistic to be used at all. If it did, it would lead to a contradiction, as demonstrated by Mayo.
While it is interesting to determine the validity of Birnbaum’s (1962) proof that the sufficiency principle (SP) and one of the versions of the conditionality principle (CP) together imply the likelihood principle (LP), I believe there is a deeper problem with the theorem. Specifically, the CP cannot be justified from a conditional inference perspective. The reasoning is as follows.
Fisher provided a number of examples showing that conditioning on an ancillary statistic was a reasonable approach. The CP is the result of extrapolating these small number of examples into a principle that requires us to always condition on an ancillary when one exists. The question is: Is this a case of unjustified extrapolation? I believe so. Consider first the intuitive argument for the mixture example. Informally, (from Giere 1977), the weak conditionality principle (WCP) claims the “irrelevance of (component) experiments not actually performed” while the LP claims “the irrelevance of outcomes not actually observed”. It is easy to see the intuitive appeal of the WCP when described this way. However, WCP could also be described informally as claiming “the irrelevance of some outcomes not actually observed.” Frequentists are concerned to use an appropriate reference set, but how do we know that conditioning on this ancillary variable provides the best reference set to use? The answer is we do not.
To see this, imagine a mixture model with two ancillaries, A and B, where A is the flip of the coin in the mixture experiment and where conditioning on A leads to much weaker inferences than conditioning on B. Ancillary B reflects a partition of the sample space across both components experiments. Thus, conditioning on B is preferred over A in the mixture model but is not available to use for conditioning in either component experiment. Not only will the WCP fail for this admittedly special case, it heralds a more-common problem. Perturb this first mixture model slightly so that A is still ancillary but B is now only approximately-ancillary. Conditioning on B will still be preferred over A even though A is the unique ancillary in the modified problem. In short, the conditionality principles cannot be justified because there can be better statistics to condition on than ancillaries. [Cox’s comment on Buehler’s (1982) ancillary paper discusses the need for forms of inference to be robust to minor perturbations of model specification. The WCP fails this test.]
Finally, as an aside, some history that tends to be overlooked in discussions about Birnbaum’s theorem. Giere (1977) reports that Birnbaum rejected the Likelihood Principle within two years of the publication of his theorem. Birnbaum abandoned the LP in favor of what he called the confidence concept in his 1969 paper.
