If the likelihood principle clashes with frequentist probability then do we discard one of them?

Question

In a comment recently posted here one commenter pointed to a blog by Larry Wasserman who points out (without any sources) that frequentist inference clashes with the likelihood principle.

The likelihood principle simply says that experiments yielding similar likelihood functions should yield similar inference.

Two parts to this question:

1. Which parts, flavour or school of frequentist inference specifically violate the likelihood principle?

2. If there is a clash, do we have to discard one or the other? If so, then which one? I will for the sake of discussion suggest that if we have to discard something then we should discard the parts of frequentist inference which clash, because Hacking and Royall have convinced me that the likelihood principle is axiomatic.

Answer 1

The part of the Frequentist approach that clashes with the likelihood principle is the theory of statistical testing (and p-value computation). It is usually highlighted by the following example.

Suppose two Frequentist want to study a biased coin, which turns 'heads' with unknown propability $p$. They suspect that it is biased towards 'tail', so they postulate the same null hypothesis $p = 1/2$ and the same alternative hypothesis $p < 1/2$.

The first statistician flips the coin until 'heads' turns up, which happens to be 6 times. The second decides to flip the coin 6 times, and obtains only one 'heads' in the last throw.

According to the model of the first statistician, the p-value is computed as follows:

$$ p(1-p)^5 + p(1-p)^6 + ... = p(1-p)^5 \frac{1}{1-p} = p(1-p)^4. $$

According to the model of the second statistician, the p-value is computed as follows:

$$ {6 \choose 1} p(1-p)^5 + {6 \choose 0} (1-p)^6 = (5p + 1)(1-p)^5. $$

Replacing $p$ by $1/2$, the first finds a p-value equal to $1/2^5 = 0.03125$, the second finds a p-value equal to $7/2 \times 1/2^5 = 0.109375$.

So, they get different results because they did different things, right? But according to the likelihood principle, they should come to the same conclusion. Briefly, the likelihood principle states that likelihood is all that matters for inference. So the clash here comes from the fact that both observations have the same likelihood, proportional to $p(1-p)^5$ (likelihood is determined up to a proportionality constant).

As far as I know, the answer to your second question is more of a debated opinion. I personally try to avoid performing tests and computing p-values for the reason above, and for others explained in this blog post.

EDIT: Now that I think about it, estimations of $p$ by confidence intervals would also differ. Actually if the models are different, the CI differ by construction.

Answer 2

I like the example by @gui11aume (+1), but it can make an impression that the difference in two $p$-values arises only due to the different stopping rules used by the two experimenters.

In fact, I believe it is a much more general phenomenon. Consider the second experimenter in @gui11aume's answer: the one who throws a coin six times and observes heads only in the last throw. The outcomes look like that: $$\mathrm{T \;\;\; T \;\;\;T \;\;\;T \;\;\;T \;\;\;H},$$ what is the $p$-value? The usual approach would be to compute the probability that a fair coin would result in one or less heads. There are $7$ possibilities out of total $64$ with one or less heads, hence the $p=7/64\approx 0.109$.

But why not take another test statistic? For example, in this experiment we observed five tails in a row. Let's take the length of the longest sequence of tails as the test statistic. There are $3$ possibilities with five or six tails in a row, hence $p=3/64\approx0.047$.

So if in this case the error rate were fixed at $\alpha=0.05$, then the choice of the test statistic can easily render the results either significant or not, and this has nothing to do with the stopping rules per se.

---

Speculative part

Now, philosophically, I would say that the frequentist choice of the test statistic is in some vague sense similar to the Bayesian choice of prior. We choose one or another test statistic because we believe that the unfair coin would behave in this or that particular way (and we want to have power to detect this behaviour). Isn't it similar to putting prior on the coin types?

If so, then the likelihood principle saying that all the evidence is in the likelihood does not clash with the $p$-values, because the $p$-value is then not only the "amount of evidence". It is "a measure of surprise", but something can only be a measure of surprise if it accounts for what we would be surprised about! The $p$-value attempts to combine in one scalar quantity both the evidence and some sort of prior expectations (as represented in the choice of the test statistic). If so, then it should not be compared to the likelihood itself, but perhaps rather to the posterior?

I would be very interested to hear some opinions about this speculative part, here or in chat.

---

Update following discussion with @MichaelLew

I am afraid that my example above missed the point of this debate. Choosing a different test statistic leads to a change in likelihood function as well. So two different $p$-values computed above correspond to two different likelihood functions, and hence cannot be an example of a "clash" between the likelihood principle and $p$-values. The beauty of the @gui11aume's example is that the likelihood function stays exactly the same, even though the $p$-values differ.

I still have to think what this means for my "speculative" part above.