# Why likelihood is not always a density function? [duplicate]

I try to self-learn Bayesian machine learning (mostly by studying Bishop and Kevin Murphy's books).

While working with formulas I was puzzled by the quote that "Note that the likelihood function is not necessary a density function".

The Bayes formula stated in terms of data $D$ and hypothesis $h$ is:

$$p(h|D) = \frac{p(D|h)p(h)}{p(D)} = \frac{likelihood \times prior}{evidence} \propto likelihood \times prior$$

I can see the previous formula simply as a formula between probabilities of some events. This looks not so useful since it does not incorporates the uncertainty, and I can agree with that.

The second interpretation, which I believe it incorporates the uncertainty is to look at the posterior probability as a probability density. In this way we might take a look at "which is the shape of the uncertainty", if I am allowed to state that. Considering the chaining of learning which happens in Bayesian learning, where posteriors might be used as prior for further learning, a normal consequence is to have the prior also as a density.

Now my question is what about the likelihood and evidence? If I am free to consider as likelihood any function of $D$ and $h$, than the evidence is simply a constant. The purpose of the evidence is diminished to a constant in order to be used for making the prior a density function. (I think so because a density function has to have its integral equal with 1). So working on this line the evidence is simply a constant which depends only on the integral of the product between likelihood and prior. If we g even further we might think that we can incorporate the evidence value in likelihood so we need no evidence there. Even the name evidence become meaningless.

On the other side if we consider likelihood a density itself, I have difficulties with the math.

Can you provide me with some insights regarding why that happens (likelihood is not necessary a density)?

• – Momo
Commented Jun 21, 2014 at 22:18
• I see that duplicated question contains in some what what I was interested for. However, until the answer of @kjetil which describes also the context, I did not had a clear intuition. I am OK with closing this question if you consider. However that question is about something else, and simply the fact that it contains in comments the answer to my question I think is not so useful for someone who ask what I asked. (I saw that question, while searching for an answer, but, obviously, I missed the answer) Commented Jun 23, 2014 at 14:10
• Perhaps it is a close call. My opinion is that it is clear from the first (also by Kjetil) and third answer, that a likelihood function for the same data can be defined up to a multiplicative constant, thus unless that constant is one at least one cannot be a density. Anyway, perhaps this answer makes it clearer, closing just stops additional answers.
– Momo
Commented Jun 23, 2014 at 15:39
• In the context of maximum likelihood estimator, it is obvious that any constant is irrelevant. My question is not closed in the MLE estimation, but in the more general Bayesian framework. Due to my innocent/stupid previous understanding I did not noticed that you really start with likelihood, and after you enrich the model with the Bayesian equation you simply switch the fixed/reference point (you switch from fixed parameter to fixed data) to estimate parameters. In the book the switch was already assumed, and in this context the likelihood is not a pdf anymore. Commented Jun 23, 2014 at 17:33

You start with a model, usually given as a family of densities (or probability mass functions). We write this as $$Y \sim f(y;\theta) ~~\text{ for \theta \in \Theta }$$ where $$\Theta$$ is the parameter space. Here, the meaning is that $$\theta$$ is an unknown parameter. For each fixed $$\theta$$ this gives a probability distribution for data $$Y$$. Now, the likelihood function is a function of $$\theta$$ for fixed $$y$$, there is nothing in the setup which could imply this to be a density function!, sometimes, by accident it "is" so (because by accident it integrates to one), but that accident has nothing to do with its logical status. Sometimes (often) it is not. An instructive example is $$Y \sim U(0, \theta) \qquad \theta > 0$$ That is, a uniform distribution. If we now observes $$Y=1$$, then the likelihood function becomes $$L(\theta) = \frac{1}{\theta} ~~ \text{for \theta > 1}$$ which has total integral $$\int_{1}^{\infty} \frac{1}{\theta}\, d\theta$$ which do not converge (that is, is $$\infty$$) so clearly is not a probability density neither can be normalized to be.