# If the likelihood principle clashes with frequentist probability then do we discard one of them?

In a comment recently posted here one commenter pointed to a blog by Larry Wasserman who points out (without any sources) that frequentist inference clashes with the likelihood principle.

The likelihood principle simply says that experiments yielding similar likelihood functions should yield similar inference.

Two parts to this question:

1. Which parts, flavour or school of frequentist inference specifically violate the likelihood principle?

2. If there is a clash, do we have to discard one or the other? If so, then which one? I will for the sake of discussion suggest that if we have to discard something then we should discard the parts of frequentist inference which clash, because Hacking and Royall have convinced me that the likelihood principle is axiomatic.

-
I have never understand why the likelihood principle should be an axiom. –  Stéphane Laurent Oct 21 '12 at 19:35
Hi, Stéphane. The problem is that Birnbaum proved that the Likelihood is equivalent to other two principles that are so natural that they should necessarily hold. We wrote a short review on this result. Here: ime.usp.br/~pmarques/papers/redux.pdf –  Zen Oct 22 '12 at 0:42
@Zen Thank you. At first glance the point I disagree with is this sentence written below the conditionality principle: "What matters is what actually happened". I should say instead "What matters is what actually happened among the issues which could have occur" (sorry if my english is not correct). That's what I claimed in my discussion with gui11aume: in a certain sense the likelihood principle claims that the design of the experiment does not matter, and I cannot agree with this point. –  Stéphane Laurent Oct 22 '12 at 5:16
@Zen Now I have read more carefully your paper. That's true that it's difficult to disagree with the conditionality principle and the invariance principle. –  Stéphane Laurent Oct 22 '12 at 9:00
LP is not that popular nowadays for practical reasons. By adopting it religiously you avoid the use of model-dependent priors such as the Jeffreys's prior, conjugate priors and hypothesis testing which can be useful in many contexts. I believe that statistics, same as physics, cannot be axiomatised in a meaningful way (although this discussion may sound like this). But it is important to identify advantages and disadvantages of different paradigms. –  user10525 Oct 24 '12 at 16:07

The part of the Frequentist approach that clashes with the likelihood principle is the theory of statistical testing (and p-value computation). It is usually highlighted by the following example.

Suppose two Frequentist want to study a biased coin, which turns 'heads' with unknown propability $p$. They suspect that it is biased towards 'tail', so they postulate the same null hypothesis $p = 1/2$ and the same alternative hypothesis $p < 1/2$.

The first statistician flips the coin until 'heads' turns up, which happens to be 6 times. The second decides to flip the coin 6 times, and obtains only one 'heads' in the last throw.

According to the model of the first statistician, the p-value is computed as follows:

$$p(1-p)^5 + p(1-p)^6 + ... = p(1-p)^5 \frac{1}{1-p} = p(1-p)^4.$$

According to the model of the second statistician, the p-value is computed as follows:

$${6 \choose 1} p(1-p)^5 + {6 \choose 0} (1-p)^6 = (5p + 1)(1-p)^5.$$

Replacing $p$ by $1/2$, the first finds a p-value equal to $1/2^5 = 0.03125$, the second finds a p-value equal to $7/2 \times 1/2^5 = 0.109375$.

So, they get different results because they did different things, right? But according to the likelihood principle, they should come to the same conclusion. Briefly, the likelihood principle states that likelihood is all that matters for inference. So the clash here comes from the fact that both observations have the same likelihood, proportional to $p(1-p)^5$ (likelihood is determined up to a proportionality constant).

As far as I know, the answer to your second question is more of a debated opinion. I personally try to avoid performing tests and computing p-values for the reason above, and for others explained in this blog post.

EDIT: Now that I think about it, estimations of $p$ by confidence intervals would also differ. Actually if the models are different, the CI differ by construction.

-
I am under the impression that the likelihood principle is obviously violated in frequentist statistics (hypothesis testing, confidence intervals) because we take into consideration the probability of each possible outcome, not only the likelihood based on the actual outcome. Right ? –  Stéphane Laurent Oct 21 '12 at 19:34
@Stéphane Laurent yes, that's also how I understand it. James Berger has a nice quote in Statistical Decision Theory and Bayesian Analysis, which says that Frequentist sometimes reject hypothesis because of data that was never observed (it sounds better, but I cannot remember it). –  gui11aume Oct 21 '12 at 19:50
Thanks, gui11aume. Am I right to interpret that as an example where the 'meaning' of P-values varies with the intent of the experimenter? I assume that is the case when P-values are interpreted as a sort of threshold false positive error rate because they would have to be uniformly distributed under the null hypothesis? Is that needed with Fisher's approach where P-values are presented as indices of the strength of evidence? –  Michael Lew Oct 21 '12 at 19:52
@Michael Lew I think that the meaning of p-values always varies with the intent of the experimenter, because you need a model and an alternative hypothesis to compute it (which are up to the experimenter). I am not sure I follow your rationale on the second part of the comment. –  gui11aume Oct 21 '12 at 19:57
(+1) This sort of discrepancies usually appear when a stopping rule is involved in one of the models. –  user10525 Oct 24 '12 at 15:44