12
$\begingroup$

While reading about likelihood, I have heard that "the exact value of any likelihood is meaningless" why?

So, because of that we may use the likelihood ratio.

So, my question is, why the exact value of likelihood is meaningless? And what is the benefit of likelihood-ratio over the likelihood?

$\endgroup$
2
  • 2
    $\begingroup$ There is no scale so some people say something like "the likelihood of $\theta$ is proportional to ..." $\endgroup$
    – Henry
    Jun 11, 2022 at 10:42
  • 2
    $\begingroup$ I was intrigued where the OP might have heard that "the exact value of any likelihood is meaningless". Quotes usually imply there is a reference. Google pointed me to A Guide on Data Analysis. $\endgroup$
    – dipetkov
    Jun 11, 2022 at 17:01

5 Answers 5

20
$\begingroup$

It's “meaningless” in the sense that it's very hard to interpret, it's just “the bigger, the better”. That is the case because the likelihood is not probability and it is calculated without normalizing the constant so the numerical value can be any non-negative number. You still can maximize the likelihood because lack of normalization doesn't matter for optimization. See other questions tagged as for more details like What is the reason that a likelihood function is not a pdf? or What is the difference between "likelihood" and "probability"?.

$\endgroup$
16
  • 3
    $\begingroup$ The likelihood is a probability (or probability density) in the data space (not the model-parameter space). Specifically, it is the probability of the data given the model parameters. The integral of the likelihood over all possible data comes to 1. In MCMC, one often works with an un-normalized version of the likelihood, and that un-normalized version is, of course, not a proper probability. $\endgroup$
    – apdnu
    Jun 12, 2022 at 12:45
  • 5
    $\begingroup$ @apdnu it's not, it's more complicated, see stats.stackexchange.com/q/2641/35989 Also you seem to assume a Bayesian perspective, where the term exists also outside it. $\endgroup$
    – Tim
    Jun 12, 2022 at 13:20
  • 1
    $\begingroup$ The likelihood is literally defined as the probability of the data, given a model. It's a probability in data-space, integrates to one in that space, is non-negative, etc. $\endgroup$
    – apdnu
    Jun 13, 2022 at 11:46
  • 1
    $\begingroup$ It sounds like your objections are more philosophical (Frequentist vs. Bayesian) than practical. Likelihood functions exactly like a probability (or probability density) in the data-space, and that is a perfectly valid interpretation of what it is. Indeed, that is exactly how it is generally defined, and any other definition is equivalent to this definition under a shift of your philosophical outlook (e.g., from Frequentist to Bayesian). $\endgroup$
    – apdnu
    Jun 13, 2022 at 11:59
  • 1
    $\begingroup$ @apdnu Likelihood functions have the parameter(s) of interest as the x-axis scale. How is that "in the data-space"? $\endgroup$ Jun 13, 2022 at 21:18
9
$\begingroup$

When we use likelihood then we are comparing probability (density) of the data given a certain hypothesis/theory.

The actual probability is not important. It can actually become extremely small. Imagine that you are testing whether a coin is a fair coin and you observe 1000 000 flips with 500 000 heads and 500 000 tails. If the coin is fair (equal head and tails probability), then the probability of this particular observation is 00.0007978844.

$\endgroup$
4
  • $\begingroup$ Can you show the proof for this here? :) $\endgroup$
    – JED HK
    Dec 28, 2022 at 10:04
  • $\begingroup$ @JEDHK proof of what? $\endgroup$ Dec 28, 2022 at 10:32
  • $\begingroup$ I mean the mathematical derivation that 1 000 000 flips with 500 000 heads and tails being 0.0007978844 $\endgroup$
    – JED HK
    Dec 28, 2022 at 16:05
  • $\begingroup$ @JEDHK I had this calculated by a computer. In R you would use dbinom(500000,1000000,0.5). Alternatively you can use the asymptotic approximation for the middle term and compute $\frac{\sqrt{2}}{\sqrt{n\pi}}$. I guess that I used the first method because the second gives me 0.0007978846 $\endgroup$ Dec 28, 2022 at 16:33
7
$\begingroup$

The likelihood function is usually taken to be the PDF viewed as as a function of parameters for known data.

For example, if I have a coin with Heads probability $\theta$ and toss it $n = 10$ times, getting $x = 3$ heads, then I can take the likelihood function to be

${n\choose x}\theta^x(1-\theta)^{n-x},$ considered as a function of $\theta.$

If I want the MLE $\hat \theta$ of $\theta,$ then I might write the likelihood function as $$f(\theta \mid x = 3)\propto \theta^3(1-\theta)^7,$$ where the symbol $\propto$ (read as "proportional to") indicates that the constant ${n\choose x} = {10\choose 3} = 120$ is omitted. The maximum of the likelihood function is at $\hat \theta = x/n = 0.3,$ whether I use or ignore the constant ${10\choose 3}.$

So the values of the likelihood function might be considered less important than its shape, which leads to the MLE $\hat\theta.$

enter image description here

$\endgroup$
2
  • 2
    $\begingroup$ It is more than that: If you saw THTTTTTHTH then the probability of that exact result would have been $\theta^3(1-\theta)^7$ while the probability of $3$ heads and $7$ tails in any order would have been ${10 \choose 3}\theta^3(1-\theta)^7$. The likelihood of $\theta$ being say $0.3$ rather than some other value does not change between these two ways of finding the probability so it is reasonable to say each is only proportional to the likelihood. $\endgroup$
    – Henry
    Jun 12, 2022 at 23:48
  • $\begingroup$ @Henry. Thanks for edit fixing typo. $\endgroup$
    – BruceET
    Jun 13, 2022 at 0:16
4
$\begingroup$

[Context]

@Henry and @JonathanLew firmly pointed out errors in my original answer, which argued that the statement "the exact value of any likelihood is meaningless" is glib and that you can't prove a logical claim about all likelihoods by providing specific examples where it's safe to compute the likelihood up to a constant (which admittedly is often the case).

Since I first posted my answer I've learned that continuous likelihoods have (theoretical) units given by 1/(units of the data) from @apdnu's answer to Units for likelihoods and probabilities.

I've come across examples where likelihood function should be be computed exactly to get the correct answer. These examples teach me that I should be careful with my likelihood calculations and don't presume that I can safely ignore normalizing constants.

And I've discovered that (a version of) this question has been asked and answered before: What does "likelihood is only defined up to a multiplicative constant of proportionality" mean in practice?

Example #1: Comparing a model with normal errors to a model with Cauchy errors

This example is from Chapter 6, Y. Pawitan In All Likelihood: Statistical Modelling And Inference Using Likelihood (2013).

We want to model Y in terms of X; there are a few unusual values in the data (outliers). We propose two models with the same mean structure E(Y) = β0 + β1X but different error structure: in one model the errors are iid Normal(0, σ2), in the other model the errors are iid Cauchy(0, γ). We fit the models by maximizing the likelihoods and next we use AIC = -2$\log$L + 2k (where k is the number of model parameters) to choose the "better" model. Both the Normal density and the Cauchy density have constant terms that are usually save to ignore: (2𝜋)-1/2 for the Normal and 𝜋-1 for the Cauchy. These are not the same constant for both models, so no parts of the likelihood functions can be dropped.

Example #2: Mixture of Bernoullis for latent class analysis

This example is from Chapter 9 of C. M. Bishop. Pattern Recognition and Machine Learning (2006).

We want to model a dataset of binary observations as a mixture of $K$ Bernoulli components with parameters $\{\mu_k\}$ and mixing proportions $\pi_k$. The log likelihood is:

$$ \ln p(\mathbf{X}|\mathbf{\mu},\mathbf{\pi}) = \sum_{n=1}^N\ln\left\{\sum_{k=1}^K\pi_kp(\mathbf{x}_n|\mathbb{\mu}_k)\right\} $$

Since there is a summation inside a logarithm, the math doesn't simplify but the maximum likelihood solution can be found with the EM algorithm.

Example #3: Bayesian $t$-test

This example is from Chapter 4 of K. P. Murphy. Machine Learning: A Probabilistic Perspective (2012).

We want to test the hypothesis $\mu > \mu_0$ for some known value of $\mu_0$. The p-value for an one-side t-test is an integral over the likelihood:

$$ \begin{aligned} p(\mu>\mu_0|\text{data}) = \int_{\mu_0}^\infty p(\mu|\text{data})d\mu \end{aligned} $$

We can't omit any terms inside the integral or we won't compute the p-value correctly.

In summary, there are both theory and examples to illustrate that the exact value of the likelihood function can be meaningful.


[Original answer, with corrections following comments]

The statement "the exact value of any likelihood is meaningless" is abstract and imprecise at the same time. So let's start with the definition of likelihood. In the spirit of this question, the definition isn't mathematically rigorous.

We take a probabilistic model f(x,θ) for data x with parameter θ.

  • As a function of the data x, f(x,θ) is a probability density/mass function. [pdf if x is continuous; pmf if x is discrete.]
  • As a function of the parameter θ, f(x,θ) is the likelihood.

It's true that the likelihood doesn't integrate to 1. Many functions don't, yet we don't conclude that their exact value is meaningless.

  • x f(x,θ) dx = 1 [replace the integral with a summation if x is discrete]
  • θ f(x,θ) dθ = constant that depends on the model f and the data x

A common theme running through the answers is that likelihood computations often simplify. The logical argument seems to go something like this: in many computations a term in the likelihood is constant or behaves like a constant so we can simplify the math by dropping that term; ergo the exact value of a likelihood function is meaningless.

However, the likelihood has more uses than maximizing it to find the MLE or performing a likelihood ratio test. And a likelihood term that can be ignored in one computation is important to keep track of in another.

$\endgroup$
7
  • 2
    $\begingroup$ But clearly you can rescale both likelihoods by the same amount without affecting the Likelihood Ratio or Bayes Factor. Suppose you toss a biased coin three times and want to consider the models of the probability of heads as $\theta_0=\frac15$ or $\theta_1=\frac45$. If you see $HHT$ it does not matter whether you say the likelihood of $\theta$ is then $\theta^2(1-\theta)$ or say ${3\choose 2}\theta^2(1-\theta)$ as you will get a ratio of $4$ in both cases. $\endgroup$
    – Henry
    Jun 11, 2022 at 21:44
  • $\begingroup$ In your $\theta_0 = 0.1$ and $\theta_1 = 0.12$ case and my $HHT$ example, it still does not matter whether you say the likelihood is $\theta^2(1-\theta)$ or ${3\choose 2}\theta^2(1-\theta)$ as you would get a ratio of $1.408$ either way. $\endgroup$
    – Henry
    Jun 12, 2022 at 0:17
  • $\begingroup$ You seem to be suggesting that if one model thinks order matters and the other that order does not matter then there is an issue over the likelihood of the latter. I am saying that since the calculations you do for the likelihoods are only meaningful up to proportionality, you can scale them to be on a comparable basis $\endgroup$
    – Henry
    Jun 12, 2022 at 0:17
  • $\begingroup$ $\theta^2(1-\theta)$ is the probability of Heads then Heads then Tails in that order. ${3\choose 2}\theta^2(1-\theta)$ is the probability of $2$ Heads and $1$ Tails in any order. It would be peculiar if your assessment of the likelihood of a particular value of $\theta$ having observed $HHT$ depends on whether you use the full observation or a sufficient statistic but, since likelihood is relative, you do need to be consistent in your choice $\endgroup$
    – Henry
    Jun 12, 2022 at 0:32
  • $\begingroup$ Not at all, and I did not say that - it is obviously important in finding a binomial probability and is not meaningless. I am saying it can be used or not in the calculation of a likelihood (so long as this is done consistently) and in that sense does not affect the likelihood of a parameter taking a particular value. Hence my original comment $\endgroup$
    – Henry
    Jun 12, 2022 at 1:08
3
$\begingroup$

An important concept that should be mentioned in the context of this discussion is that of the Likelihood principle (See also Berger & Wolpert's book). The likelihood principle, which is one of the foundations of Bayesian statistics, states that all the evidence relevant to a model parameter $\theta$ is contained in the likelihood function $\mathcal L(\theta | x) = f_X(x|\theta)$.

The precise statement of the likelihood principle is that if two experiments about the same parameters $\theta$ are proportional to each other, namely if

$$f_X(x|\theta) = cf_Y(y|\theta)$$

with $c$ some positive constant, then the evidence on $\theta$ from the experiments is identical. In this sense, the likelihood is only meaningful (as evidence) up to a normalizing constant.

Indeed in Bayesian statistics those two likelihoods will lead to the same posterior probability of $\theta$ (given the same prior), hence Bayesian statistics 'automatically' respects the likelihood principle.

Frequentists statistics in general violates the likelihood principle, so discussion of the meaning of likelihood from a frequentist perspective is somewhat meaningless by itself. But practically most if not all frequentist uses of the likelihood function are via quantities (such as likelihood ratios or the maximum likelihood estimator) that are also invariant under scaling,

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.