2
$\begingroup$

In Bayes' Theorem (particularly in the classification problem), we are given an input $x$ and we want to know what class $C_k$ it belongs to. Hence $p(C_k|x) = \dfrac{p(x|C_k)p(C_k)}{p(x)}$. Here, $p(C_k)$ is known as the prior distribution. (I understood why this is named 'prior'.)

What I don't understand is why $p(x|C_k)$ is named likelihood function.

It doesn't help that in ordinary language, 'likelihood' is used interchangeably with 'probability'.

Finally, does 'likelihood' have anything to do with the maximum likelihood estimator likelihood function?

$\endgroup$
1
  • $\begingroup$ There is a paper that I read some time ago about this topic, but since I am not very acquainted with old players of statistical theory, it was a kind of difficulty to get the correct perspective. Maybe it can be helpful for you. I would like to know a simple answer as well: economics.soton.ac.uk/staff/aldrich/ident.pdf $\endgroup$ Feb 28, 2020 at 2:33

1 Answer 1

2
$\begingroup$

This is a good question. I'll answer them in reverse.

does 'likelihood' have anything to do with the maximum likelihood estimator likelihood function?

Yes! If you have a sequence of iid random variables $x_1, \dots, x_n$ then we call the following function the likelihood.

$$ \mathcal{L}(\theta;x) = \prod_{i} f(x_i; \theta) = p(x\vert \theta) $$

Here, the density of the $x$, $f$, is parameterized by $\theta$. Since the rvs are iid, their joint probability is just their product, so the likelihood is a probability distribution. We write that distribution as $p(x\vert \theta)$

What I don't understand is why 𝑝(𝑥|𝐶𝑘) is named likelihood function.

Let's rewrite Bayes theorem in terms of $x$ and $\theta$.

$$ p(\theta \vert x) = \dfrac{p(x\vert \theta) p(\theta)}{p(\theta)} $$

We see that the $p(x\vert \theta)$ shows up in the numerator. This $p(x\vert \theta)$ and the $p(x\vert \theta)$ above are the very same. Hence, we call that part the likelihood.

$\endgroup$
4
  • 1
    $\begingroup$ Well, the distribution function is a function of $x$ given $\theta$, but the likelihood function is a function of $\theta$ given $x$. The character sequences used to write them are usually the same, but conceptually they are different. $\endgroup$
    – jbowman
    Feb 28, 2020 at 2:37
  • $\begingroup$ @jbowman It just depends on what you are considering as fixed and what you are considering as a variable. The likelihood is what I've written. Whether you think of theta as fixed or x as fixed will lead you to either the left or right hand equality. The difference is the one you impose on it. $\endgroup$ Feb 28, 2020 at 3:26
  • $\begingroup$ I appreciate the insight. But then I have a follow up question ... So the MLE likelihood estimator (since you are multiplying) does not become a valid pdf because the result does not lie between 0 to 1. Yet in the Bayes' Theorem case, $p(x|\theta)$ clearly is a valid pdf, no? $\endgroup$
    – cgo
    Feb 28, 2020 at 7:44
  • 1
    $\begingroup$ That all results from abuse of notation. The function $x \mapsto p(x | \theta)$, for fixed $\theta$, is a valid pdf. But the likelihood, $\theta \mapsto p(x | \theta)$ for fixed $x$, is not a valid pdf in general. $\endgroup$ Feb 28, 2020 at 11:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.