# If the likelihood principle clashes with frequentist probability then do we discard one of them?

In a comment recently posted here one commenter pointed to a blog by Larry Wasserman who points out (without any sources) that frequentist inference clashes with the likelihood principle.

The likelihood principle simply says that experiments yielding similar likelihood functions should yield similar inference.

Two parts to this question:

1. Which parts, flavour or school of frequentist inference specifically violate the likelihood principle?

2. If there is a clash, do we have to discard one or the other? If so, then which one? I will for the sake of discussion suggest that if we have to discard something then we should discard the parts of frequentist inference which clash, because Hacking and Royall have convinced me that the likelihood principle is axiomatic.

• I have never understand why the likelihood principle should be an axiom. – Stéphane Laurent Oct 21 '12 at 19:35
• Hi, Stéphane. The problem is that Birnbaum proved that the Likelihood is equivalent to other two principles that are so natural that they should necessarily hold. We wrote a short review on this result. Here: ime.usp.br/~pmarques/papers/redux.pdf – Zen Oct 22 '12 at 0:42
• @Zen Thank you. At first glance the point I disagree with is this sentence written below the conditionality principle: "What matters is what actually happened". I should say instead "What matters is what actually happened among the issues which could have occur" (sorry if my english is not correct). That's what I claimed in my discussion with gui11aume: in a certain sense the likelihood principle claims that the design of the experiment does not matter, and I cannot agree with this point. – Stéphane Laurent Oct 22 '12 at 5:16
• @Zen Now I have read more carefully your paper. That's true that it's difficult to disagree with the conditionality principle and the invariance principle. – Stéphane Laurent Oct 22 '12 at 9:00
• LP is not that popular nowadays for practical reasons. By adopting it religiously you avoid the use of model-dependent priors such as the Jeffreys's prior, conjugate priors and hypothesis testing which can be useful in many contexts. I believe that statistics, same as physics, cannot be axiomatised in a meaningful way (although this discussion may sound like this). But it is important to identify advantages and disadvantages of different paradigms. – user10525 Oct 24 '12 at 16:07

The part of the Frequentist approach that clashes with the likelihood principle is the theory of statistical testing (and p-value computation). It is usually highlighted by the following example.

Suppose two Frequentist want to study a biased coin, which turns 'heads' with unknown propability $p$. They suspect that it is biased towards 'tail', so they postulate the same null hypothesis $p = 1/2$ and the same alternative hypothesis $p < 1/2$.

The first statistician flips the coin until 'heads' turns up, which happens to be 6 times. The second decides to flip the coin 6 times, and obtains only one 'heads' in the last throw.

According to the model of the first statistician, the p-value is computed as follows:

$$p(1-p)^5 + p(1-p)^6 + ... = p(1-p)^5 \frac{1}{1-p} = p(1-p)^4.$$

According to the model of the second statistician, the p-value is computed as follows:

$${6 \choose 1} p(1-p)^5 + {6 \choose 0} (1-p)^6 = (5p + 1)(1-p)^5.$$

Replacing $p$ by $1/2$, the first finds a p-value equal to $1/2^5 = 0.03125$, the second finds a p-value equal to $7/2 \times 1/2^5 = 0.109375$.

So, they get different results because they did different things, right? But according to the likelihood principle, they should come to the same conclusion. Briefly, the likelihood principle states that likelihood is all that matters for inference. So the clash here comes from the fact that both observations have the same likelihood, proportional to $p(1-p)^5$ (likelihood is determined up to a proportionality constant).

As far as I know, the answer to your second question is more of a debated opinion. I personally try to avoid performing tests and computing p-values for the reason above, and for others explained in this blog post.

EDIT: Now that I think about it, estimations of $p$ by confidence intervals would also differ. Actually if the models are different, the CI differ by construction.

• I am under the impression that the likelihood principle is obviously violated in frequentist statistics (hypothesis testing, confidence intervals) because we take into consideration the probability of each possible outcome, not only the likelihood based on the actual outcome. Right ? – Stéphane Laurent Oct 21 '12 at 19:34
• @Stéphane Laurent yes, that's also how I understand it. James Berger has a nice quote in Statistical Decision Theory and Bayesian Analysis, which says that Frequentist sometimes reject hypothesis because of data that was never observed (it sounds better, but I cannot remember it). – gui11aume Oct 21 '12 at 19:50
• Thanks, gui11aume. Am I right to interpret that as an example where the 'meaning' of P-values varies with the intent of the experimenter? I assume that is the case when P-values are interpreted as a sort of threshold false positive error rate because they would have to be uniformly distributed under the null hypothesis? Is that needed with Fisher's approach where P-values are presented as indices of the strength of evidence? – Michael Lew Oct 21 '12 at 19:52
• (+1) This sort of discrepancies usually appear when a stopping rule is involved in one of the models. – user10525 Oct 24 '12 at 15:44
• @Scortchi Actually I was mistaken to think that one of the P-values points to the correct likelihood function and the other not: they both point to the same likelihood function which does present the evidence relevant to the probability of heads. You should ignore the last two sentences of my previous comment. (I can't edit it, can I?) – Michael Lew Feb 19 '15 at 19:57

I like the example by @gui11aume (+1), but it can make an impression that the difference in two $p$-values arises only due to the different stopping rules used by the two experimenters.

In fact, I believe it is a much more general phenomenon. Consider the second experimenter in @gui11aume's answer: the one who throws a coin six times and observes heads only in the last throw. The outcomes look like that: $$\mathrm{T \;\;\; T \;\;\;T \;\;\;T \;\;\;T \;\;\;H},$$ what is the $p$-value? The usual approach would be to compute the probability that a fair coin would result in one or less heads. There are $7$ possibilities out of total $64$ with one or less heads, hence the $p=7/64\approx 0.109$.

But why not take another test statistic? For example, in this experiment we observed five tails in a row. Let's take the length of the longest sequence of tails as the test statistic. There are $3$ possibilities with five or six tails in a row, hence $p=3/64\approx0.047$.

So if in this case the error rate were fixed at $\alpha=0.05$, then the choice of the test statistic can easily render the results either significant or not, and this has nothing to do with the stopping rules per se.

### Speculative part

Now, philosophically, I would say that the frequentist choice of the test statistic is in some vague sense similar to the Bayesian choice of prior. We choose one or another test statistic because we believe that the unfair coin would behave in this or that particular way (and we want to have power to detect this behaviour). Isn't it similar to putting prior on the coin types?

If so, then the likelihood principle saying that all the evidence is in the likelihood does not clash with the $p$-values, because the $p$-value is then not only the "amount of evidence". It is "a measure of surprise", but something can only be a measure of surprise if it accounts for what we would be surprised about! The $p$-value attempts to combine in one scalar quantity both the evidence and some sort of prior expectations (as represented in the choice of the test statistic). If so, then it should not be compared to the likelihood itself, but perhaps rather to the posterior?

I would be very interested to hear some opinions about this speculative part, here or in chat.

### Update following discussion with @MichaelLew

I am afraid that my example above missed the point of this debate. Choosing a different test statistic leads to a change in likelihood function as well. So two different $p$-values computed above correspond to two different likelihood functions, and hence cannot be an example of a "clash" between the likelihood principle and $p$-values. The beauty of the @gui11aume's example is that the likelihood function stays exactly the same, even though the $p$-values differ.

I still have to think what this means for my "speculative" part above.

• Interesting thoughts. Yes, I agree that there need be no conflict between the LP and P-values as long as the P-values are not interpreted as evidence in the same way as the likelihood function is. The likelihood function contains the evidence relevant to the parameter of interest given the statistical model. When you change the test statistic you change the model, so the likelihood function for your alternative model will (well, may) differ from the likelihood function for the original. – Michael Lew Feb 18 '15 at 23:09
• Michael, I am not sure what exactly "statistical model" means, but isn't a coin with heads probability $p$ already a model? How does changing the test statistic change the model? – amoeba Feb 18 '15 at 23:11
• Apart from that, I found this question because I was re-reading your "To P or not to P" paper (and googled "likelihood principle"). I generally like the paper, but I got completely confused by the section 4.4. You write that the p-values should not be "adjusted" by taking stopping rules into account; but I don't see any adjustments in the formulas 5-6. What would "unadjusted" p-values be? Do you mean that one of them is adjusted and another one not? If so, which one, and why not vice versa? – amoeba Feb 18 '15 at 23:13
• The statistical model is often ignored or tacitly assumed to be invariant. However, for the coins it includes a fixed unknown probability of heads, a random selection of observations, and, for the head out of trials test statistic, the binomial distribution of possible outcomes. I don't know what the distribution of the outcomes is for the tails in a row test statistic but I suspect it is different. Even if it is the same, the model which has your test statistic is not the same model as the original and so the likelihood function can be different even though it contains all of the evidence. – Michael Lew Feb 18 '15 at 23:15
• I'm nearly finished a complete reworking of that paper. It is relevant to this discussion but not yet ready for submission. (Is this chat?) – Michael Lew Feb 18 '15 at 23:16