Why likelihood is not always a density function? [duplicate]

Question

I try to self-learn Bayesian machine learning (mostly by studying Bishop and Kevin Murphy's books).

While working with formulas I was puzzled by the quote that "Note that the likelihood function is not necessary a density function".

The Bayes formula stated in terms of data $D$ and hypothesis $h$ is:

$$ p(h|D) = \frac{p(D|h)p(h)}{p(D)} = \frac{likelihood \times prior}{evidence} \propto likelihood \times prior$$

I can see the previous formula simply as a formula between probabilities of some events. This looks not so useful since it does not incorporates the uncertainty, and I can agree with that.

The second interpretation, which I believe it incorporates the uncertainty is to look at the posterior probability as a probability density. In this way we might take a look at "which is the shape of the uncertainty", if I am allowed to state that. Considering the chaining of learning which happens in Bayesian learning, where posteriors might be used as prior for further learning, a normal consequence is to have the prior also as a density.

Now my question is what about the likelihood and evidence? If I am free to consider as likelihood any function of $D$ and $h$, than the evidence is simply a constant. The purpose of the evidence is diminished to a constant in order to be used for making the prior a density function. (I think so because a density function has to have its integral equal with 1). So working on this line the evidence is simply a constant which depends only on the integral of the product between likelihood and prior. If we g even further we might think that we can incorporate the evidence value in likelihood so we need no evidence there. Even the name evidence become meaningless.

On the other side if we consider likelihood a density itself, I have difficulties with the math.

Can you provide me with some insights regarding why that happens (likelihood is not necessary a density)?

Answer 1

You start with a model, usually given as a family of densities (or probability mass functions). We write this as $$ Y \sim f(y;\theta) ~~\text{ for $\theta \in \Theta$ } $$ where $\Theta$ is the parameter space. Here, the meaning is that $\theta$ is an unknown parameter. For each fixed $\theta$ this gives a probability distribution for data $Y$. Now, the likelihood function is a function of $\theta$ for fixed $y$, there is nothing in the setup which could imply this to be a density function!, sometimes, by accident it "is" so (because by accident it integrates to one), but that accident has nothing to do with its logical status. Sometimes (often) it is not. An instructive example is $$ Y \sim U(0, \theta) \qquad \theta > 0 $$ That is, a uniform distribution. If we now observes $Y=1$, then the likelihood function becomes $$ L(\theta) = \frac{1}{\theta} ~~ \text{for $\theta > 1$} $$ which has total integral $\int_{1}^{\infty} \frac{1}{\theta}\, d\theta $ which do not converge (that is, is $\infty$) so clearly is not a probability density neither can be normalized to be.