Bayes' Theorem (likelihood function): What's in a name?

In Bayes' Theorem (particularly in the classification problem), we are given an input $$x$$ and we want to know what class $$C_k$$ it belongs to. Hence $$p(C_k|x) = \dfrac{p(x|C_k)p(C_k)}{p(x)}$$. Here, $$p(C_k)$$ is known as the prior distribution. (I understood why this is named 'prior'.)

What I don't understand is why $$p(x|C_k)$$ is named likelihood function.

It doesn't help that in ordinary language, 'likelihood' is used interchangeably with 'probability'.

Finally, does 'likelihood' have anything to do with the maximum likelihood estimator likelihood function?

• There is a paper that I read some time ago about this topic, but since I am not very acquainted with old players of statistical theory, it was a kind of difficulty to get the correct perspective. Maybe it can be helpful for you. I would like to know a simple answer as well: economics.soton.ac.uk/staff/aldrich/ident.pdf Feb 28, 2020 at 2:33

This is a good question. I'll answer them in reverse.

does 'likelihood' have anything to do with the maximum likelihood estimator likelihood function?

Yes! If you have a sequence of iid random variables $$x_1, \dots, x_n$$ then we call the following function the likelihood.

$$\mathcal{L}(\theta;x) = \prod_{i} f(x_i; \theta) = p(x\vert \theta)$$

Here, the density of the $$x$$, $$f$$, is parameterized by $$\theta$$. Since the rvs are iid, their joint probability is just their product, so the likelihood is a probability distribution. We write that distribution as $$p(x\vert \theta)$$

What I don't understand is why π(π₯|πΆπ) is named likelihood function.

Let's rewrite Bayes theorem in terms of $$x$$ and $$\theta$$.

$$p(\theta \vert x) = \dfrac{p(x\vert \theta) p(\theta)}{p(\theta)}$$

We see that the $$p(x\vert \theta)$$ shows up in the numerator. This $$p(x\vert \theta)$$ and the $$p(x\vert \theta)$$ above are the very same. Hence, we call that part the likelihood.

• Well, the distribution function is a function of $x$ given $\theta$, but the likelihood function is a function of $\theta$ given $x$. The character sequences used to write them are usually the same, but conceptually they are different. Feb 28, 2020 at 2:37
• @jbowman It just depends on what you are considering as fixed and what you are considering as a variable. The likelihood is what I've written. Whether you think of theta as fixed or x as fixed will lead you to either the left or right hand equality. The difference is the one you impose on it. Feb 28, 2020 at 3:26
• I appreciate the insight. But then I have a follow up question ... So the MLE likelihood estimator (since you are multiplying) does not become a valid pdf because the result does not lie between 0 to 1. Yet in the Bayes' Theorem case, $p(x|\theta)$ clearly is a valid pdf, no?
– cgo
Feb 28, 2020 at 7:44
• That all results from abuse of notation. The function $x \mapsto p(x | \theta)$, for fixed $\theta$, is a valid pdf. But the likelihood, $\theta \mapsto p(x | \theta)$ for fixed $x$, is not a valid pdf in general. Feb 28, 2020 at 11:21