6
$\begingroup$

I have trouble discerning conceptually between these two notions. I am aware of their formal relations, proprieties and what not, but I just can't wrap my head around what they "mean", if that even makes sense. I have checked some answers, such as this one, but I think none of these addresses my problem.

So, given some data $\mathcal{D}$ and parameters $\theta$,

$$ p(\theta|\mathcal{D})=\frac{p(\mathcal{D}|\theta)p(\theta)}{p(\mathcal{D})} $$

I have learned that the posterior is "the probability of $\theta$ being the statistical parameter underlying $\mathcal{D}$". And the likelihood is "the likelihood of $\theta$ having generated $\mathcal{D}$". In my head, these two notions are exactly the same. So how can I distinguish them?

Improve this question
$\endgroup$
  • $\begingroup$ I would just say it like the conditional formula says it: the posterior is the probability of getting $\theta$ given some data $D$, the likelihood is the probability of getting $D$ if the parameter $\theta$ is true. $\endgroup$ – user2974951 Oct 3 '19 at 10:22
  • 1
    $\begingroup$ I don't think that thinking of $P(D|\theta)$, as you have, "the likelihood of $\theta$ having generated D" is a useful way of thinking about it. I think of it as "if we knew that $\theta$ took a particular value, what would the probability of observing the data we have be?" That is different to "given the data we have, what do we think the true value of $\theta$ is?" $\endgroup$ – gazza89 Oct 3 '19 at 10:23
  • $\begingroup$ @gazza89 this is actually how I thought of this before getting into ML and bayesian probability. I used to think of p(x|y) as simply conditional probabilities, but now I hear that the likelihood is not a p.d.f. and this kinda confuses me. If it's just the probability of D given $\theta$, then why is it not a pdf? $\endgroup$ – olinarr Oct 3 '19 at 10:34
  • 1
    $\begingroup$ The likelihood is a pdf, it's just normalised w.r.t all possible data outcomes, and the posterior is a pdf, but it's normalised w.r.t all possible parameter values $\endgroup$ – gazza89 Oct 3 '19 at 10:39
3
$\begingroup$

First, to have a posterior distribution for $\theta$, $\theta$ must be (modeled as) a random variable. For the likelihood function that is not necessary. So this is deeper than the comment (by @gazza89) saying

The likelihood is a pdf, it's just normalised w.r.t all possible data outcomes, and the posterior is a pdf, but it's normalised w.r.t all possible parameter values

Even if the likelihood was (or could be normalized too, it is not always possible) to integrate to one, that is not enough to make it into a pdf (probability density function). A pdf must be the pdf of some random variable and if $\theta$ is not modeled as a random variable, then it cannot have a pdf, period.

So conceptually it is quite clear-cut:

  • If $\theta$ is a random variable (probably in some Bayesian model) then it can have a posterior, and it can have a likelihood function. Even if those two functions should be numerically equal (as for instance if the prior is uniform), they are distict mathematical entities.

  • If $\theta$ is not (modeled as) a random variable the problem does not arise, only the likelihood function can be defined.

You say

And the likelihood is "the likelihood of θ having generated D"

but that is not a good way of thinking about it. Likelihood tells you the probability (under the model ...) of the data if data was generated according to that value of $\theta$. There is no "probability of $\theta$" involved. To get that you need some extra assumptions, a Bayesian probability model.

To understand likelihood better see all the answers to Maximum Likelihood Estimation (MLE) in layman terms.

Improve this answer
$\endgroup$
1
$\begingroup$

To put simply, likelihood is "the likelihood of $\theta$ having generated $\mathcal{D}$" and posterior is essentially "the likelihood of $\theta$ having generated $\mathcal{D}$" further multiplied by the prior distribution of $\theta$.

If the prior distribution is flat (or non-informative), likelihood is exactly the same as posterior.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.