Bayes' Theorem (likelihood function): What's in a name?

Question

In Bayes' Theorem (particularly in the classification problem), we are given an input $x$ and we want to know what class $C_k$ it belongs to. Hence $p(C_k|x) = \dfrac{p(x|C_k)p(C_k)}{p(x)}$. Here, $p(C_k)$ is known as the prior distribution. (I understood why this is named 'prior'.)

What I don't understand is why $p(x|C_k)$ is named likelihood function.

It doesn't help that in ordinary language, 'likelihood' is used interchangeably with 'probability'.

Finally, does 'likelihood' have anything to do with the maximum likelihood estimator likelihood function?

Answer 1

This is a good question. I'll answer them in reverse.

does 'likelihood' have anything to do with the maximum likelihood estimator likelihood function?

Yes! If you have a sequence of iid random variables $x_1, \dots, x_n$ then we call the following function the likelihood.

$$ \mathcal{L}(\theta;x) = \prod_{i} f(x_i; \theta) = p(x\vert \theta) $$

Here, the density of the $x$, $f$, is parameterized by $\theta$. Since the rvs are iid, their joint probability is just their product, so the likelihood is a probability distribution. We write that distribution as $p(x\vert \theta)$

What I don't understand is why 𝑝(𝑥|𝐶𝑘) is named likelihood function.

Let's rewrite Bayes theorem in terms of $x$ and $\theta$.

$$ p(\theta \vert x) = \dfrac{p(x\vert \theta) p(\theta)}{p(\theta)} $$

We see that the $p(x\vert \theta)$ shows up in the numerator. This $p(x\vert \theta)$ and the $p(x\vert \theta)$ above are the very same. Hence, we call that part the likelihood.