While reading about likelihood, I have heard that "the exact value of any likelihood is meaningless" why?
So, because of that we may use the likelihood ratio.
So, my question is, why the exact value of likelihood is meaningless? And what is the benefit of likelihood-ratio over the likelihood?
It's “meaningless” in the sense that it's very hard to interpret, it's just “the bigger, the better”. That is the case because the likelihood is not probability and it is calculated without normalizing the constant so the numerical value can be any non-negative number. You still can maximize the likelihood because lack of normalization doesn't matter for optimization. See other questions tagged as likelihood for more details like What is the reason that a likelihood function is not a pdf? or What is the difference between "likelihood" and "probability"?.
When we use likelihood then we are comparing probability (density) of the data given a certain hypothesis/theory.
The actual probability is not important. It can actually become extremely small. Imagine that you are testing whether a coin is a fair coin and you observe 1000 000 flips with 500 000 heads and 500 000 tails. If the coin is fair (equal head and tails probability), then the probability of this particular observation is 00.0007978844.
dbinom(500000,1000000,0.5). Alternatively you can use the asymptotic approximation for the middle term and compute $\frac{\sqrt{2}}{\sqrt{n\pi}}$. I guess that I used the first method because the second gives me 0.0007978846
$\endgroup$
Commented
Dec 28, 2022 at 16:33
The likelihood function is usually taken to be the PDF viewed as as a function of parameters for known data.
For example, if I have a coin with Heads probability $\theta$ and toss it $n = 10$ times, getting $x = 3$ heads, then I can take the likelihood function to be
${n\choose x}\theta^x(1-\theta)^{n-x},$ considered as a function of $\theta.$
If I want the MLE $\hat \theta$ of $\theta,$ then I might write the likelihood function as $$f(\theta \mid x = 3)\propto \theta^3(1-\theta)^7,$$ where the symbol $\propto$ (read as "proportional to") indicates that the constant ${n\choose x} = {10\choose 3} = 120$ is omitted. The maximum of the likelihood function is at $\hat \theta = x/n = 0.3,$ whether I use or ignore the constant ${10\choose 3}.$
So the values of the likelihood function might be considered less important than its shape, which leads to the MLE $\hat\theta.$
[Context]
@Henry and @JonathanLew firmly pointed out errors in my original answer, which argued that the statement "the exact value of any likelihood is meaningless" is glib and that you can't prove a logical claim about all likelihoods by providing specific examples where it's safe to compute the likelihood up to a constant (which admittedly is often the case).
Since I first posted my answer I've learned that continuous likelihoods have (theoretical) units given by 1/(units of the data) from @apdnu's answer to Units for likelihoods and probabilities.
I've come across examples where likelihood function should be be computed exactly to get the correct answer. These examples teach me that I should be careful with my likelihood calculations and don't presume that I can safely ignore normalizing constants.
And I've discovered that (a version of) this question has been asked and answered before: What does "likelihood is only defined up to a multiplicative constant of proportionality" mean in practice?
Example #1: Comparing a model with normal errors to a model with Cauchy errors
This example is from Chapter 6, Y. Pawitan In All Likelihood: Statistical Modelling And Inference Using Likelihood (2013).
We want to model Y in terms of X; there are a few unusual values in the data (outliers). We propose two models with the same mean structure E(Y) = β0 + β1X but different error structure: in one model the errors are iid Normal(0, σ2), in the other model the errors are iid Cauchy(0, γ). We fit the models by maximizing the likelihoods and next we use AIC = -2$\log$L + 2k (where k is the number of model parameters) to choose the "better" model. Both the Normal density and the Cauchy density have constant terms that are usually save to ignore: (2𝜋)-1/2 for the Normal and 𝜋-1 for the Cauchy. These are not the same constant for both models, so no parts of the likelihood functions can be dropped.
Example #2: Mixture of Bernoullis for latent class analysis
This example is from Chapter 9 of C. M. Bishop. Pattern Recognition and Machine Learning (2006).
We want to model a dataset of binary observations as a mixture of $K$ Bernoulli components with parameters $\{\mu_k\}$ and mixing proportions $\pi_k$. The log likelihood is:
$$ \ln p(\mathbf{X}|\mathbf{\mu},\mathbf{\pi}) = \sum_{n=1}^N\ln\left\{\sum_{k=1}^K\pi_kp(\mathbf{x}_n|\mathbb{\mu}_k)\right\} $$
Since there is a summation inside a logarithm, the math doesn't simplify but the maximum likelihood solution can be found with the EM algorithm.
Example #3: Bayesian $t$-test
This example is from Chapter 4 of K. P. Murphy. Machine Learning: A Probabilistic Perspective (2012).
We want to test the hypothesis $\mu > \mu_0$ for some known value of $\mu_0$. The p-value for an one-side t-test is an integral over the likelihood:
$$ \begin{aligned} p(\mu>\mu_0|\text{data}) = \int_{\mu_0}^\infty p(\mu|\text{data})d\mu \end{aligned} $$
We can't omit any terms inside the integral or we won't compute the p-value correctly.
In summary, there are both theory and examples to illustrate that the exact value of the likelihood function can be meaningful.
[Original answer, with corrections following comments]
The statement "the exact value of any likelihood is meaningless" is abstract and imprecise at the same time. So let's start with the definition of likelihood. In the spirit of this question, the definition isn't mathematically rigorous.
We take a probabilistic model f(x,θ) for data x with parameter θ.
It's true that the likelihood doesn't integrate to 1. Many functions don't, yet we don't conclude that their exact value is meaningless.
A common theme running through the answers is that likelihood computations often simplify. The logical argument seems to go something like this: in many computations a term in the likelihood is constant or behaves like a constant so we can simplify the math by dropping that term; ergo the exact value of a likelihood function is meaningless.
However, the likelihood has more uses than maximizing it to find the MLE or performing a likelihood ratio test. And a likelihood term that can be ignored in one computation is important to keep track of in another.
An important concept that should be mentioned in the context of this discussion is that of the Likelihood principle (See also Berger & Wolpert's book). The likelihood principle, which is one of the foundations of Bayesian statistics, states that all the evidence relevant to a model parameter $\theta$ is contained in the likelihood function $\mathcal L(\theta | x) = f_X(x|\theta)$.
The precise statement of the likelihood principle is that if two experiments about the same parameters $\theta$ are proportional to each other, namely if
$$f_X(x|\theta) = cf_Y(y|\theta)$$
with $c$ some positive constant, then the evidence on $\theta$ from the experiments is identical. In this sense, the likelihood is only meaningful (as evidence) up to a normalizing constant.
Indeed in Bayesian statistics those two likelihoods will lead to the same posterior probability of $\theta$ (given the same prior), hence Bayesian statistics 'automatically' respects the likelihood principle.
Frequentists statistics in general violates the likelihood principle, so discussion of the meaning of likelihood from a frequentist perspective is somewhat meaningless by itself. But practically most if not all frequentist uses of the likelihood function are via quantities (such as likelihood ratios or the maximum likelihood estimator) that are also invariant under scaling,
