Is there an example where two different defensible tests with proportional likelihoods would lead one to markedly different (and equally defensible) inferences, for instance, where the p-values are order of magnitudes far apart, but the power to alternatives is similar?

All the examples I see are very silly, comparing a binomial with a negative binomial, where the p-value of the first is 7% and of the second 3%, which are "different" only insofar one is making binary decisions on arbitrary thresholds of significance such as 5% (which, by the way, is a pretty low standard for inference) and do not even bother to look at power. If I change the threshold for 1%, for instance, both lead to the same conclusion.

I've never seen an example where it would lead to markedly different and defensible inferences. Is there such an example?

I'm asking because I've seen so much ink spent on this topic, as if the Likelihood Principle is something fundamental in the foundations of statistical inference. But if the best example one has are silly examples like the one above, the principle seems completely inconsequential.

Thus, I'm looking for a very compelling example, where if one does not follow the LP the weight of evidence would be overwhelmingly pointing in one direction given one test, but, in a different test with proportional likelihood, the weight of evidence would be overwhelmingly pointing in an opposite direction, and both conclusions look sensible.

Ideally, one could demonstrate we can have arbitrarily far apart, yet sensible, answers, such as tests with $p =0.1$ versus $p= 10^{-10}$ with proportional likelihoods and equivalent power to detect the same alternative.

PS: Bruce's answer does not address the question at all.

  • 5
    $\begingroup$ When performing significance testing, one can always change the decision by changing the threshold. Could you therefore explain what you mean by "markedly," "silly," or "compelling"? BTW, you seem to be reading the Wikipedia article. $\endgroup$
    – whuber
    Nov 23, 2018 at 20:18
  • 2
    $\begingroup$ Welcome to CV, @statslearner. Can you give an example of one or more specific approaches to inference that do not use the likelihood principle which you would like to see contrasted? $\endgroup$
    – Alexis
    Nov 23, 2018 at 20:20
  • 1
    $\begingroup$ @whuber ideally I would like to see that you can construct arbitrarily different answers such as, if you want to use p-values, something like $p=0.5$ versus $p=10^{-5}$, and both computations would still seem defensible. $\endgroup$
    – user227843
    Nov 23, 2018 at 20:21
  • 3
    $\begingroup$ I cannot follow that comment because $p=10^5$ makes no sense. Regardless, have you considered just changing the numbers given in the Wikipedia example? $\endgroup$
    – whuber
    Nov 23, 2018 at 20:22
  • 6
    $\begingroup$ The significant difference with practical implications is the processing of stopping rules: under the LP they do not matter, outside the LP they do. Check Berger & Wolpert (1987) for details. $\endgroup$
    – Xi'an
    Nov 23, 2018 at 20:52

5 Answers 5


Think about a hypothetical situation when a point null hypothesis is true but one keeps sampling until $p<0.05$ (this will always happen sooner or later, i.e. it will happen with probability 1) and then decides to stop the trial and reject the null. This is an admittedly extreme stopping rule but consider it for the sake of the argument.

This moronic procedure will have 100% Type I error rate, but there is nothing wrong with it according to the Likelihood Principle.

I'd say this does count as "really" mattering. You can of course choose any $\alpha$ in this argument. Bayesians can use a fixed cut-off on Bayes factor if they please. The same logic applies. The main lesson here is that you cannot adhere to LP and have an error rate guarantee. There is no free lunch.

  • 4
    $\begingroup$ I was thinking of this example as well. But I did not mention it because it is indeed moronic. But actually, it is what happens in practice indirectly and informally. $\endgroup$ Dec 4, 2018 at 21:40
  • 1
    $\begingroup$ What are the 2 statistics and their likelihood in your example? In the neg. binomial vs binomial case we have: 1) statistics 1, number of trials until 3 heads, likelihood neg binomial; 2) statistics 2, number of heads in n trials, likehood binomail. In your example, I don't see what the two statistics are and if they have proportional likelihoods. $\endgroup$
    – user227843
    Dec 5, 2018 at 19:14
  • 1
    $\begingroup$ In your example probably it would be "number of trials until p<0.05" which I hardly doubt it is proportional to the binomial, so I'm not sure your example is valid, Amoeba. $\endgroup$
    – user227843
    Dec 5, 2018 at 19:17
  • 1
    $\begingroup$ I don't think the likelihood principle says "there is nothing wrong with it." The likelihood principle filters out bad procedures. The fact that the procedure does not obey the likelihood principle is not the same as it being endorsed by the likelihood principle. A Bayesian analysis of this sequential testing problem, which of course does obey the likelihood principle, has perfectly fine properties, because it will not implement the "moronic" procedure you describe. $\endgroup$
    – guy
    Dec 5, 2018 at 23:28
  • 3
    $\begingroup$ @amoeba consider $\theta \sim N(0,\tau^{-1})$ under the alternative or $\theta = 0$ under the null, with $Y_i \sim N(\theta,1)$. It is easy to show that the log of the Bayes factor is roughly $\frac 1 2 [\log(\tau / n) + Z_n^2]$ where $Z_n$ is the usual $Z$ test statistic. Rejecting when the Bayes factor is larger than $1$ is then equivalent to rejecting when $|Z_n| > O(\sqrt{\log n})$. Under the null, this is not guaranteed to happen in the sequential testing setting (c.f. the law of iterated logarithm); hence, the Bayesian procedure will not fall victim to the problem you described. $\endgroup$
    – guy
    Dec 7, 2018 at 23:11

Disclaimer: I believe this answer is at the core of the entire argument, so it worth discussion, but I haven't fully explored the issue. As such, I welcome corrections, refinements and comments.

The most important aspect is in regards to sequentially collected data. For example, suppose you observed binary outcomes, and you saw 10 success and 5 failures. The likelihood principle says that you should come to the same conclusion about the probability of success, regardless of whether you collected data until you had 10 successes (negative binomial) or ran 15 trials, of which 10 were successes (binomial).

Why is this of any importance?

Because according to the likelihood principle (or at least, a certain interpretation of the it), it's totally fine to let the data influence when you're going to stop collecting data, without having to alter your inference tools.

Conflict with Sequential Methods

The idea that using your data to decide when to stop collecting data without altering your inferential tools flies completely in the face of traditional sequential analysis methods. The classic example of this is with methods used in clinical trials. In order to reduce potential exposure to harmful treatments, data is often analyzed at intermediate times before the analysis is done. If the trial hasn't finished yet, but the researchers already have enough data to conclude that the treatment works or is harmful, medical ethics tells us we should stop the trial; if the treatment works, it is ethical to stop the trial and start making the treatment available to non-trial patients. If it is harmful, it is more ethical to stop so that we stop exposing trial patients to a harmful treatment.

The problem is now we've started to do multiple comparisons, so we've increased our Type I error rate if we do not adjust our methods to account for the multiple comparisons. This isn't quite the same as traditional multiple comparisons problems, as it's really multiple partial comparisons (i.e., if we analyze the data once with 50% of the data collected and once with 100%, these two samples clearly are not independent!), but in general the more comparisons we do, the more we need to change our criteria for rejecting the null hypothesis to preserve the type I error rate, with more comparisons planned requiring more evidence to reject the null.

This puts clinical researchers in a dilemma; do you want to frequently check your data, but then increase your required evidence to reject the null, or do you want to infrequently check your data, increasing your power but potentially not acting in an optimal manner in regards to medical ethics (i.e., may delay product to market or expose patients unnecessarily long to harmful treatment).

It is my (perhaps mistaken) understanding that the likelihood principle appears to tell us that it doesn't matter how many times we check the data, we should make the same inference. This basically says that all the approaches to sequential trial design are completely unnecessary; just use the likelihood principle and stop once you've collected enough data to make a conclusion. Since you don't need to alter your inference methods to adjust for the number of analyses you've prepared, there is no trade off dilemma between number of times checked and power. Bam, whole field of sequential analysis is solved (according to this interpretation).

Personally, what is very confusing about this to me is that a fact that is well know in the sequential design field, but fairly subtle, is that the likelihood of the final test statistic is largely altered by the stopping rule; basically, the stopping rules increase the probability in a discontinuous manner at the stopping points. Here is a plot of such a distortion; the dashed line is the PDF of the final test statistic under the null if data is only analyzed after all data is collected, while the solid line gives you the distribution under the null of the test statistic if you check the data 4 times with a given rule.

With that said, it's my understanding that the likelihood principle seems to imply that we can throw out all we know about Frequentist sequential design and forget about how many times we analyze our data. Clearly, the implications of this, especially for the field of clinical designs, is enormous. However, I haven't wrapped my mind around how they justify ignoring how stopping rules alter the likelihood of the final statistic.

Some light discussion can be found here, mostly on the final slides.

  • 2
    $\begingroup$ +1. I find it conceptually easier to think about a hypothetical situation when the null hypothesis is true but one keeps sampling until $p<0.05$ (this wall always happen sooner or later, i.e. it will happen with probability 1) and then decides to stop the trial. This moronic procedure will have 100% Type I error rate, even though it complies with the LP. $\endgroup$
    – amoeba
    Dec 4, 2018 at 20:20
  • $\begingroup$ @amoeba: I agree that your example is pretty straightforward (+1). The goal of my answer is to emphasize why is there even a discussion. I think that answer is that if the implications and interpretations of the LP were correct, it would mean that clinical trials would no longer have to chose between maximal power and unnecessary exposure, which would be an absolutely huge gain. In general it would also free researchers from needing to guess proper sample size in advance, which greatly improve the utility of statistical tests. $\endgroup$
    – Cliff AB
    Dec 4, 2018 at 23:38
  • $\begingroup$ Well, I think the whole framework of frequentist testing is inconsistent with the LP, and that's just how it is. One uses frequentist testing if one wants a guarantee on the error rates. Turns out that this is inconsistent with LP. See also Lindley's paradox and all that. Well, tough. I used to be excited about these matters, but now I am not anymore. There is no free lunch; one has to make some choices. Note that a lot of Bayesian procedures violate LP as well. $\endgroup$
    – amoeba
    Dec 5, 2018 at 8:18
  • $\begingroup$ "the likelihood of the final test statistic is largely altered by the stopping rule" The pdf is changed, and also the likelihood (but only by a constant), but you may still end up with a likelihood functions that are the same up to a constant of proportionality. E.g. the binomial distribution and the negative binomial distribution for $k$ successes and $n$ trials have both a likelihood $\mathcal{L}(p|n,k)$ that is proportional to $\propto p^kp^{n-k}$ $\endgroup$ Dec 5, 2018 at 10:33

Outline of LR tests for exponential data.

Let $X_1, X_2, \dots, X_n$ be a random sample from $\mathsf{Exp}(\text{rate} =\lambda),$ so that $E(X_i) = \mu = 1/\lambda.$ For $x > 0,$ the density function is $f(x) = \lambda e^{-\lambda x}$ and the CDF is $F(x) = 1 - e^{-\lambda x}.$

1. Test statistic is sample minimum.

Let $V = X_{(1)} = \min_n (X_i).$ Then $V \sim \mathsf{Exp}(n\lambda).$ As an outline of the proof, $$P(V > v) = P(X_1 > v, \dots, X_n > v) = \left[e^{-\lambda v}\right]^n= e^{-n\lambda v},$$ so that $P(V \le v) = 1 - e^{-n\lambda v},$ for $v > 0.$

To test $H_9:\mu \le \mu_0$ against $H_a: \mu > \mu_0,$ at level $\alpha = 5\%,$ we regard $V$ as a single observation from its exponential distribution. We find that the log likelihood ratio indicates rejection when $V > c,$ where $P(V > c\, |\, \mu = \mu_0) = 0.05.$

For the specific case in which $n = 100$ and $\mu_0 =10,\, \lambda_0 = 0.1,$ we have exponential rate $10 = n/\mu_0 = 100/10 = 10,$ so that $c = 0.2295$ from R, where the exponential distribution is parameterized by the rate.

 qexp(.95, 10)
 [1] 0.2995732
 1 - pexp(0.2996, 10)
 [1] 0.04998662

Accordingly, the power against the alternative $\mu_a = 100$ (rate $n/\mu_a = 1)$ is about 74%.

1 - pexp(0.2996, 1)
[1] 0.7411146

2. Test statistic is the sample mean.

Oxford U. class notes (second page) show that the likelihood ratio test of $H_0: \mu \le \mu_0$ against $H_0: \mu > \mu_0$ at the 5% level of significance rejects for $\bar X > c,$ where $P(\bar X > c\, |\, \mu = \mu_0) = 0.5.$ Furthermore, one can show using moment generating functions that $\bar X \sim \mathsf{Gamma}(n, n\lambda).$

For the specific case in which $n = 100$ and $\mu_0 =10,\, \lambda_0 = 0.1,$ we have $\bar X \sim \mathsf{Gamma}(100, 10),$ so that $c = 11.7.$

qgamma(.95, 100, 10)
[1] 11.69971
1 - pgamma(11.7, 100, 10)
[1] 0.04997338

Accordingly, power against the alternative $\mu_a = 14$ is about 95.6%.

1 - pgamma(11.7, 100, 100/14)
[1] 0.9562513

Clearly, for purposes of testing hypotheses about the exponential mean $\mu,$ the information in the sufficient statistic $\bar X$ is much greater than the information in the sample minimum.

  • $\begingroup$ I don't think this address the question at all.Are the two likelihoods proportional? You first need to show the likelihood of the two experiments are proportional, otherwise the likelihood principle does not apply. Second, in this example the two tests lead to the same conclusion, so it's even more underwhelming than the example of the binomial versus negative binomial. $\endgroup$
    – user227843
    Nov 24, 2018 at 8:59
  • $\begingroup$ I just checked the document, the likelihoods are not proportional, since the first likelihood has $v$ in the exponent and the other has $\sum x_i$, thus the likelihood principle should not apply here, it's fine for the two tests to lead to different conclusions according to the likelihood principle. $\endgroup$
    – user227843
    Nov 24, 2018 at 9:04
  • 2
    $\begingroup$ Bruce, just to clarify what the liklihood principle states: it says that if you have two experiments where the likelihoods differ only by a constant, then you should derive the same conclusion from them. This happens in the binomial versus negative binomial case, where they differ only in the binomial coefficient part (constant). Your example shows two tests where their likelihoods do not differ only by a constant, so the LP does not apply. $\endgroup$
    – user227843
    Nov 24, 2018 at 9:08
  • $\begingroup$ @statslearner2 the likelihood function for observing a sample $x_1,...,x_n$ is: $$f(x_1,...,x_n) = \prod_{i=1}^n \lambda e^{-\lambda x_i}$$ This is the same whether you select the minimum or the mean as a criteria to perform the test. The violation that occurs here can be seen as the type in which the definition of 'extreme cases' is different and the integration to compute the p-value is done differently. $\endgroup$ Dec 3, 2018 at 13:22

Violation by different pdf functions $f(x,\theta)$ and $g(x,\theta)$

This case will be an example of 'violation' because the probability distribution functions $f(x,\theta)$ $g(x,\theta)$ are intrinsically different. Even when $f$ and $g$, differ, they may relate to the likelihood principle because at fixed measurement $x$ they give the same functions of $\theta$ up to scaling. The difference, opens up a possibility for "violations".

The coin flip with or without optional stopping rule

The coin flip with or without optional stopping rule is a typical example, the pdf is binomial or negative binomial which are different pdf functions and lead to different calculation of p-values, and confidence intervals, but they lead to the same likelihood functions for fixed sample/measurement (up to scaling).

$$\begin{array}{rcrl} f_{\text{Negative Binomial}}(n|k,p) &=& {{n-1}\choose{k-1}}&p^k(1-p)^{n-k} \\ f_{\text{Binomial}}(k|n,p) &=& {{n}\choose{k}}&p^k(1-p)^{n-k} \end{array}$$

More extreme example

Consider some measurement of $X$ which is distributed as

$$\mathcal{L}(\theta | x) = f(x|\theta) = \begin{cases} 0 & \text{ if } \quad x < 0 \\a & \text{ if }\quad 0 \geq x < 1 \\ (1-a) \theta \exp(-\theta (x-1)) & \text{ if }\quad x \geq 1 \end{cases}$$

where $a$ is some known parameter that depends on the type of experiment, and $\theta$ is some parameter that may be unknown and could be inferred from the measurement $x$.

For any given $x$ and $a$ the likelihood function is proportional to the same function that is independent from $a$:

  • If $x<1$ then $\mathcal{L}(\theta | x) \propto 1$
  • If $x\geq 1$ then $\mathcal{L}(\theta | x) \propto \theta \exp(-\theta (x-1))$

But, albeit the same likelihood function, the p-value can vary widely depending on the experiment (ie the value of $a $). For instance when you measure $x=2$ and test $H_0:\theta = 1$ against $H_0:\theta < 1$ then the p-value is

$$P(X>2|\theta = 1) = \frac{(1-a)}{\exp(1)} $$

Intuition: The reason for violation in these cases is that p-values and hypothesis tests are not solely based on the likelihood function for the particular observed value $x$.

The p-value is not calculated from the likelihood $f(θ|x)$ with $x$ fixed, but with the pdf $f(x|θ)$ with $θ$ fixed which is a different slice. Confidence intervals, p-value, and hypothesis tests, are different things than the information from likelihood ratios.

p-values are not really evidence: The p-value relates to type I error which is a measure that relates to an ensemble of measurements rather than to a single measurement. This type I error or p-value is not the same as 'evidential meaning' from Birnbaums 'foundations of statistical evidence'. This relates a lot to the problems with p-values and scientist searching for outcomes solely with statistical significance rather than important effects.

Do we need examples where inferences are markedly different? The extreme case is a contrived example. Such a case, or anything with a similar extreme difference, is of course not occurring easily in practice. It is more often the case that the difference will be small such as in the cases that you refer to as silly.

To ask for examples where the likelihood principle 'really matters', or where two different inferences lead to extremely different results, is a bit of a loaded question. At least when the intention for this question relates to some philosophical argument. It is a loaded question because it presupposes that principles that matter should lead to extremely varying results. In many practical cases the results are however small (in terms of different p-values less than an order). I believe that this is not a strange for two different, but both plausible, methods to result in more or less similar results. I would consider the likelihood principle not to be 'less violated' when the differences are only small.

  • $\begingroup$ Regarding Case 1: I think choosing a different test statistic can (should?) be seen as changing the likelihood function. $\endgroup$
    – amoeba
    Dec 4, 2018 at 20:35
  • 2
    $\begingroup$ @MartijnWeterings yes it is choosing a different test statistics, what matters is the likelihood of the statistics, not of the data. Otherwise I can take a sequence of 100 flips and compute several statsistics: number of runs of heads, number of alternations of heads and tails. None of this violates the LP. $\endgroup$
    – user227843
    Dec 5, 2018 at 19:15
  • $\begingroup$ You need to pick two statistics that will have proportional likelihoods, such as the number of trials until 3 success or the number of successes in n trials etc. $\endgroup$
    – user227843
    Dec 5, 2018 at 19:25

Here is an example adapted from Statistical decision theory and Bayesian analysis by James O. Berger (Second edition page 29).

Say that two species of wasps can be distinguished by the number of notches on the wings (call this $x$) and by the number of black rings around the abdomen (call this $y$). The distribution of the characters in the two species (labelled $H_0$ and $H_1$) are as follows:

Table adapted from Statistical decision theory and Bayesian analysis by James O. Berger.

Say that we find a specimen with 1 notch on the wings and 1 ring around the abdomen. The weight of evidence if 100 times bigger in favor of $H_1$ against $H_0$ for both characters.

Now if someone wanted to set up a test for $H_0$ at 5% level, the decision rule would be for the first character “accept $H_0$ if there is 1 notch on the wing, otherwise reject it”, and for the second character “accept $H_0$ if there are 3 rings around the abdomen, otherwise reject it”. There are many other possibilities, but these ones are most powerful tests at this level. Yet, they lead to different conclusions for both characters.

Note: one could of course set up a test with the rule “accept $H_0$ if there are 1 or 3 rings around the abdomen, otherwise reject it”. The question is whether we prefer a test at 5% level with type II risk 0, or a test at 4.9% level with type II risk 0.00001. The difference is so small that we would probably not care, but as I understand it, this is the core of the argument for the likelihood principle: it is not a good idea to make the result depend on something that seems irrelevant.

The likelihood functions are proportional, and yet the p-value of $x = 1$ is 0.95, and that of $y = 1$ is 0.001 (assuming that we reject $H_0$ with events of the form $y \leq \alpha$). It is obvious from the structure of the table that I could have chosen any number smaller than 0.001. Also, the type II risk of the rejection is 0, so it looks like there is nothing “wrong” here.

Still, I admit that this example is somewhat contrived and not completely honest because it plays with the difficulty of arranging tests with discrete data. One could find equivalent examples with continuous data but they would be even more contrived. I agree with the OP that the likelihood principle has almost no practical value; I interpret it as a principle to guarantee some consistency within the theory.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy