1
$\begingroup$

I know that similar questions have already been answered on this platform but none of them were really answering my specific question which is the following:

Bayes' theorem arises solely by rearranging the multiplicative law of probability: $$p(\theta|x)p(x) = p(x|\theta)p(\theta)$$ $$ p(\theta|x) = \frac{p(x|\theta)p(\theta)}{p(x)}$$

Hence, all the quantities involved are proper pmfs or pdfs. However, I constantly read that the likelihood in Bayes theorem wouldnt be a proper probability (pmf or pdf) since it is not normalized to one. How is that possible?

I understand the concept of the likelihood function $L(\theta|x)=p(x|\theta)$ in MLE and why it is not a pdf (or pmf) since it holds the random variable x fixed and varies the parameter $\theta$. However, this cannot be used in Bayes theorem, since Bayes theorem requires that the quantities involved are pdfs (or pmfs) otherwise it would be mathematically wrong. So which mistake am I making or what do I not know about the likelihood in Bayes theorem?

Here https://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/readings/MIT18_05S14_Reading11.pdf is a numerical example where the likelihoods indeed do not add up to 1 in Bayes' theorem but I do not understand how this is possible since they should be probabilities and hence should add up to 1.

Improve this question
$\endgroup$
19
  • 2
    $\begingroup$ I marked your question as a duplicate of another one that seems to answer it. If it doesn't, please tell us why? TL;DR in MLE is not a conditional probability because $\theta$ is not a random variable, in Bayesian setting it is. $\endgroup$
    – Tim ♦
    Feb 10 '20 at 21:08
  • $\begingroup$ Hi, thank you for your reply. Yes I understand why the MLE (likelihood function L) is not a conditional probability. But the issue is, that a lot of authors state, that you would use that likelihood function L as well in Bayes theorem. For example in the link I shared they used a numerical example (table on page 3) where indeed the likelihoods do not add up to 1. And the lecturer emphasizes that fact in point 5 on page 4. In fact, he states on p.2 "Likelihood: (This is the same likelihood we used for the MLE." when he talks about the likelihood in Bayes theorem. $\endgroup$
    – guest1
    Feb 11 '20 at 15:17
  • $\begingroup$ So it seems that many authors and lecturers seem to use the likelihood function L in Bayes theorem which is nonesensical exactly because of the fact that you stated: That in Bayes theorem it has to be a conditional probabiliy distribution where $\theta$ and $x$ are random variables $\endgroup$
    – guest1
    Feb 11 '20 at 15:18
  • $\begingroup$ Similarly, Gelman writes in his book on page 7: "The second term in this expression, p(x|θ), is taken here as a function of θ, not of x", when he talks about Bayes' theorem. But that again would be wrong right? Since the likelihood as in Bayes theorem should be a distribution over random variable x conditioned on random variable $\theta$ $\endgroup$
    – guest1
    Feb 11 '20 at 15:21
  • $\begingroup$ I cannot comment on that, because I don't know what you are referring to, but in Bayesian context parameters and data are always considered as random variables. Otherwise you cannot apply Bayes theorem. $\endgroup$
    – Tim ♦
    Feb 11 '20 at 15:21
4
$\begingroup$

Not sure whether I understand precisely what you don't understand. My impression is that it just confuses you that one can speak about $p(x|\theta)$ as both a "proper pmf/pdf" (if interpreted as function over $x$) and a likelihood (if interpreted as function over $\theta$).

The formula gives you the value for $p(\theta|x)$ for fixed values of $x$ and $\theta$, and for this it doesn't matter whether $p(x|\theta)$ is interpreted as function over $x$ or over $\theta$. So one can say that there are only proper pmfs/pdfs in the formula, but (interpreting differently what $p(x|\theta)$ is a function over) also that there's the likelihood in it, which is not a pdf/pmf. (One can also say that $p(\theta|x)$ and $p(x|\theta)$ are both functions of both $\theta$ and $x$, and again there's some freedom to focus on $x$ or $\theta$ when interpreting them.)

Actually for $p(\theta|x)$ to become a proper pdf/pmf over $\theta$ given $x$, $p(x|\theta)$ must be a pmf/pdf over $x$ for given $\theta$, which is just what it is. It does not have to be a pmf/pdf over $\theta$, which it isn't.

Improve this answer
$\endgroup$
10
  • $\begingroup$ Okay I think your comment has made it a bit more clear to me. So to reiterate it in my own words: The likelihood L and $p(x|\theta)$ are given by the same formula. The difference is just what is interpreted as random variable and what as parameter. Or more precisely, with respect to which quantity it is taken to be a function/probability density. However, what is still not clear to me: If i plug in that formula but I interpret it as a likelihood function, then it doesn't have the properties of a probability density anymore and hence Bayes theorem could not be applied anymore $\endgroup$
    – guest1
    Mar 3 '20 at 11:30
  • $\begingroup$ Because if $p(x|\theta)$ is not interpreted as the conditional probability of x given $\theta$ but as function of $\theta$ (treated as parameters not as random variable) given fixed x, then the transformation properties are completely different. Hence, if I multiply it with, e.g., the prior probability density, then e.g. the multiplicative law of probability cannot be used. In particular, $p(x|\theta) p(\theta) = p(x, \theta)$ but $L(\theta) p(\theta) \neq p(x, \theta)$ $\endgroup$
    – guest1
    Mar 3 '20 at 11:32
  • $\begingroup$ The mathematical properties of $p(x|\theta)$ and its theory don't depend on whether you call it a likelihood or a conditional pmf/pdf. It's still $p(x|\theta)$, so if you call it $L(\theta)$ for given $x$, then still $L(\theta)p(\theta)=p(x,\theta)$ (although this would be somewhat confusing notation because it drops the dependence on $x$). It's mathematics, and mathematics is governed by well-defined mathematical objects, not by interpretations. $\endgroup$
    – Christian Hennig
    Mar 3 '20 at 13:52
  • $\begingroup$ Hmm, but if x is not treated as a random variable, the multiplicative law of probability just doesnt apply does it? And the likelihood $L(\theta)$ is treating neither x as a random variable nor $\theta$. Hence, the multiplicative law of probability does not apply. So it certainly depends on how a quantity is interpreted. Just as an example: If I interpret some quantity y as a random variable then it has properties like a mean, a variance etc. If i treat y just like a parameter then it is just a real number. So it does depend on how i interpret (or more precisely: define) a quantity $\endgroup$
    – guest1
    Mar 3 '20 at 15:18
  • $\begingroup$ I can't follow you. If $L(\theta)=p(x|\theta)$, then $p(x,\theta)=p(x|\theta)p(\theta)=L(\theta)p(\theta)$. This of course relies on considering a specific $x$ that is dropped in the notation of $L(\theta)$. Note that $L(\theta)$ is a value, whereas $L(\bullet)$ is a (likelihood) function. $L(\theta)$ is its value at specific $\theta$, the very same value as $p(x|\theta)$, so just because you write it $L(\theta)$, it doesn't actually "know" or change its meaning according to the fact that it is written as a function of $\theta$. $\endgroup$
    – Christian Hennig
    Mar 3 '20 at 15:31
4
$\begingroup$

If integrated the conditional probability you would get

$$ \int_\Theta p(\theta|x)d\theta = 1,$$ as expected - the posterior is a proper probability distribution, where I define proper to be that the integral over the parameter space is 1 and not just finite. But in many cases a probability distribution is in practice a product of bounded, positive functions, each individually not a proper probability distribution. In Bayes' theorem, the posterior is

$$\frac{p(x|\theta)p(\theta)}{p(x)},$$

but this puts no requirements on $p(x|\theta)$ or $p(\theta)$ individually: $p(x|\theta)$ is a probability distribution in $x$, but it is just a function in $\theta$. Thus, the integral $$\int_\Theta p(x|\theta) d\theta \neq 1$$ in many interesting cases.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.