12
$\begingroup$

While reading about likelihood, I have heard that "the exact value of any likelihood is meaningless" why?

So, because of that we may use the likelihood ratio.

So, my question is, why the exact value of likelihood is meaningless? And what is the benefit of likelihood-ratio over the likelihood?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ There is no scale so some people say something like "the likelihood of $\theta$ is proportional to ..." $\endgroup$
    – Henry
    Jun 11 at 10:42
  • 2
    $\begingroup$ I was intrigued where the OP might have heard that "the exact value of any likelihood is meaningless". Quotes usually imply there is a reference. Google pointed me to A Guide on Data Analysis. $\endgroup$
    – dipetkov
    Jun 11 at 17:01

5 Answers 5

Reset to default
18
$\begingroup$

It's “meaningless” in the sense that it's very hard to interpret, it's just “the bigger, the better”. That is the case because the likelihood is not probability and it is calculated without normalizing the constant so the numerical value can be any non-negative number. You still can maximize the likelihood because lack of normalization doesn't matter for optimization. See other questions tagged as for more details like What is the reason that a likelihood function is not a pdf? or What is the difference between "likelihood" and "probability"?.

Improve this answer
$\endgroup$
16
  • 3
    $\begingroup$ The likelihood is a probability (or probability density) in the data space (not the model-parameter space). Specifically, it is the probability of the data given the model parameters. The integral of the likelihood over all possible data comes to 1. In MCMC, one often works with an un-normalized version of the likelihood, and that un-normalized version is, of course, not a proper probability. $\endgroup$
    – apdnu
    Jun 12 at 12:45
  • 4
    $\begingroup$ @apdnu it's not, it's more complicated, see stats.stackexchange.com/q/2641/35989 Also you seem to assume a Bayesian perspective, where the term exists also outside it. $\endgroup$
    – Tim ♦
    Jun 12 at 13:20
  • 1
    $\begingroup$ The likelihood is literally defined as the probability of the data, given a model. It's a probability in data-space, integrates to one in that space, is non-negative, etc. $\endgroup$
    – apdnu
    Jun 13 at 11:46
  • 1
    $\begingroup$ It sounds like your objections are more philosophical (Frequentist vs. Bayesian) than practical. Likelihood functions exactly like a probability (or probability density) in the data-space, and that is a perfectly valid interpretation of what it is. Indeed, that is exactly how it is generally defined, and any other definition is equivalent to this definition under a shift of your philosophical outlook (e.g., from Frequentist to Bayesian). $\endgroup$
    – apdnu
    Jun 13 at 11:59
  • 1
    $\begingroup$ @apdnu Likelihood functions have the parameter(s) of interest as the x-axis scale. How is that "in the data-space"? $\endgroup$
    – Michael Lew
    Jun 13 at 21:18
8
$\begingroup$

When we use likelihood then we are comparing probability (density) of the data given a certain hypothesis/theory.

The actual probability is not important. It can actually become extremely small. Imagine that you are testing whether a coin is a fair coin and you observe 1000 000 flips with 500 000 heads and 500 000 tails. If the coin is fair (equal head and tails probability), then the probability of this particular observation is 00.0007978844.

Improve this answer
$\endgroup$
5
$\begingroup$

The likelihood function is usually taken to be the PDF viewed as as a function of parameters for known data.

For example, if I have a coin with Heads probability $\theta$ and toss it $n = 10$ times, getting $x = 3$ heads, then I can take the likelihood function to be

${n\choose x}\theta^x(1-\theta)^{n-x},$ considered as a function of $\theta.$

If I want the MLE $\hat \theta$ of $\theta,$ then I might write the likelihood function as $$f(\theta \mid x = 3)\propto \theta^3(1-\theta)^7,$$ where the symbol $\propto$ (read as "proportional to") indicates that the constant ${n\choose x} = {10\choose 3} = 120$ is omitted. The maximum of the likelihood function is at $\hat \theta = x/n = 0.3,$ whether I use or ignore the constant ${10\choose 3}.$

So the values of the likelihood function might be considered less important than its shape, which leads to the MLE $\hat\theta.$

enter image description here

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ It is more than that: If you saw THTTTTTHTH then the probability of that exact result would have been $\theta^3(1-\theta)^7$ while the probability of $3$ heads and $7$ tails in any order would have been ${10 \choose 3}\theta^3(1-\theta)^7$. The likelihood of $\theta$ being say $0.3$ rather than some other value does not change between these two ways of finding the probability so it is reasonable to say each is only proportional to the likelihood. $\endgroup$
    – Henry
    Jun 12 at 23:48
  • $\begingroup$ @Henry. Thanks for edit fixing typo. $\endgroup$
    – BruceET
    Jun 13 at 0:16
3
$\begingroup$

An important concept that should be mentioned in the context of this discussion is that of the Likelihood principle (See also Berger & Wolpert's book). The likelihood principle, which is one of the foundations of Bayesian statistics, states that all the evidence relevant to a model parameter $\theta$ is contained in the likelihood function $\mathcal L(\theta | x) = f_X(x|\theta)$.

The precise statement of the likelihood principle is that if two experiments about the same parameters $\theta$ are proportional to each other, namely if

$$f_X(x|\theta) = cf_Y(y|\theta)$$

with $c$ some positive constant, then the evidence on $\theta$ from the experiments is identical. In this sense, the likelihood is only meaningful (as evidence) up to a normalizing constant.

Indeed in Bayesian statistics those two likelihoods will lead to the same posterior probability of $\theta$ (given the same prior), hence Bayesian statistics 'automatically' respects the likelihood principle.

Frequentists statistics in general violates the likelihood principle, so discussion of the meaning of likelihood from a frequentist perspective is somewhat meaningless by itself. But practically most if not all frequentist uses of the likelihood function are via quantities (such as likelihood ratios or the maximum likelihood estimator) that are also invariant under scaling,

Improve this answer
$\endgroup$
3
$\begingroup$

[Context]

@Henry and @JonathanLew firmly pointed out errors in my original answer, which tried to argue that the statement "the exact value of any likelihood is meaningless" is glib and that you can't prove a logical claim about all likelihoods by providing specific examples where it's safe to compute the likelihood up to a constant.

Since I first posted my answer I've learned that continuous likelhoods have (theoretical) units given by 1/(units of the data) from @apdnu's answer to Units for likelihoods and probabilities.

I also searched for concrete examples where likelihood function has to be computed exactly to get the correct answer.

Example #1: Mixture of Bernoullis for latent class analysis

This example is from Chapter 9 of C. M. Bishop. Pattern Recognition and Machine Learning (2006).

We want to model a dataset of binary observations as a mixture of $K$ Bernoulli components with parameters $\{\mu_k\}$ and mixing proportions $\pi_k$. The log likelihood is:

$$ \ln p(\mathbf{X}|\mathbf{\mu},\mathbf{\pi}) = \sum_{n=1}^N\ln\left\{\sum_{k=1}^K\pi_kp(\mathbf{x}_n|\mathbb{\mu}_k)\right\} $$

Since there is a summation inside a logarithm, the math doesn't simplify but the maximum likelihood solution can be found with the EM algorithm.

Example #2: Bayesian $t$-test

This example is from Chapter 4 of K. P. Murphy. Machine Learning: A Probabilistic Perspective (2012).

We want to test the hypothesis $\mu > \mu_0$ for some known value of $\mu_0$. The p-value for an one-side t-test is an integral over the likelihood

$$ \begin{aligned} p(\mu>\mu_0|\text{data}) = \int_{\mu_0}^\infty p(\mu|\text{data})d\mu \end{aligned} $$

We can't omit any terms inside the integral or we won't compute the p-value correctly.

In summary, there are both theory and examples to illustrate that the exact value of the likelihood function can be meaningful.

[Original answer, with corrections following comments]

The statement "the exact value of any likelihood is meaningless" is abstract and imprecise at the same time. So let's start with the definition of likelihood. In the spirit of this question, the definition isn't mathematically rigorous.

We take a probabilistic model f(x,θ) for data x with parameter θ.

  • As a function of the data x, f(x,θ) is a probability density/mass function. [pdf if x is continuous; pmf if x is discrete.]
  • As a function of the parameter θ, f(x,θ) is the likelihood.

It's true that the likelihood doesn't integrate to 1. Many functions don't, yet we don't conclude that their exact value is meaningless.

  • ∫x f(x,θ) dx = 1 [replace the integral with a summation if x is discrete]
  • ∫θ f(x,θ) dθ = constant that depends on the model f and the data x

A common theme running through the answers is that likelihood computations often simplify. The logical argument seems to go something like this: in many computations a term in the likelihood is constant or behaves like a constant so we can simplify the math by dropping that term; ergo the exact value of a likelihood function is meaningless.

However, the likelihood has more uses than maximizing it to find the MLE or performing a likelihood ratio test. And a likelihood term that can be ignored in one computation is important to keep track of in another.

Improve this answer
$\endgroup$
7
  • 2
    $\begingroup$ But clearly you can rescale both likelihoods by the same amount without affecting the Likelihood Ratio or Bayes Factor. Suppose you toss a biased coin three times and want to consider the models of the probability of heads as $\theta_0=\frac15$ or $\theta_1=\frac45$. If you see $HHT$ it does not matter whether you say the likelihood of $\theta$ is then $\theta^2(1-\theta)$ or say ${3\choose 2}\theta^2(1-\theta)$ as you will get a ratio of $4$ in both cases. $\endgroup$
    – Henry
    Jun 11 at 21:44
  • $\begingroup$ In your $\theta_0 = 0.1$ and $\theta_1 = 0.12$ case and my $HHT$ example, it still does not matter whether you say the likelihood is $\theta^2(1-\theta)$ or ${3\choose 2}\theta^2(1-\theta)$ as you would get a ratio of $1.408$ either way. $\endgroup$
    – Henry
    Jun 12 at 0:17
  • $\begingroup$ You seem to be suggesting that if one model thinks order matters and the other that order does not matter then there is an issue over the likelihood of the latter. I am saying that since the calculations you do for the likelihoods are only meaningful up to proportionality, you can scale them to be on a comparable basis $\endgroup$
    – Henry
    Jun 12 at 0:17
  • $\begingroup$ $\theta^2(1-\theta)$ is the probability of Heads then Heads then Tails in that order. ${3\choose 2}\theta^2(1-\theta)$ is the probability of $2$ Heads and $1$ Tails in any order. It would be peculiar if your assessment of the likelihood of a particular value of $\theta$ having observed $HHT$ depends on whether you use the full observation or a sufficient statistic but, since likelihood is relative, you do need to be consistent in your choice $\endgroup$
    – Henry
    Jun 12 at 0:32
  • $\begingroup$ Not at all, and I did not say that - it is obviously important in finding a binomial probability and is not meaningless. I am saying it can be used or not in the calculation of a likelihood (so long as this is done consistently) and in that sense does not affect the likelihood of a parameter taking a particular value. Hence my original comment $\endgroup$
    – Henry
    Jun 12 at 1:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.