4
$\begingroup$

I learned that Bayes theorem was defined as follows :

$$p(\theta\mid y)=\frac{p(y\mid\theta)p(\theta)}{p(y)}$$

But then today I came across definition with likelihood:

$$p(\theta\mid y)=\frac{L(\theta\mid y)p(\theta)}{p(y)} = \frac{L(\theta\mid y) p(\theta)}{\int L(\theta\mid y) p(\theta) d\theta }$$

What is the link between the two?

Improve this question
$\endgroup$
2
$\begingroup$

$L(\theta|y) = p(y|\theta)$. I assume that $y$ is the observation here, and we are inferring the value of the parameter $\theta$, Thus, $p(y|\theta)$ can be viewed as a function $L$ over the (unknown) variables/parameters $\theta$.

For the denominator, $p(y) = \int p(y,\theta)d\theta = \int p(y|\theta)p(\theta)d\theta = \int L(\theta|y)p(\theta)d\theta$.

Improve this answer
$\endgroup$
2
$\begingroup$

The second formula is wrong: the outside parts are equal to each other, but the middle part is merely proportional to (and not necessarily equal to) the outside parts. The likelihood is defined by $L(\theta \mid y) = k(y) p(y \mid \theta) \propto p(y \mid \theta)$ where $k$ is some constant-of-proportionality that does not depend on $\theta$. This means you have:

$$p(\theta \mid y) = \frac{p(y \mid \theta) p(\theta)}{p(y)} = \frac{k(y) L(\theta \mid y) p(\theta)}{p(y)} \propto \frac{L(\theta \mid y) p(\theta)}{p(y)}.$$

Using the law of total probability you also have $p(y) = \int p(y \mid \theta) p(\theta) d\theta$ which gives:

$$p(\theta \mid y) = \frac{p(y \mid \theta) p(\theta)}{p(y)} = \frac{k(y) L(\theta \mid y) p(\theta)}{k(y) \int L(\theta \mid y) p(\theta) d\theta} = \frac{L(\theta \mid y) p(\theta)}{\int L(\theta \mid y) p(\theta) d\theta}.$$

In the special case where $k(y) = 1$ you have $L(\theta \mid y) = p(y \mid \theta)$ and so in this case you get the second equation you specified. However, it is common when using likelihood functions to use a constant-of-proportionality that effectively removes multiplicative terms that do not depend on $\theta$.

Improve this answer
$\endgroup$
  • $\begingroup$ It is wrong if they define it as unnormalized, but I've seen it defined as $L(\theta | y) = p(y | \theta)$ in some texts... $\endgroup$ – Tim♦ Aug 27 '18 at 9:18
  • $\begingroup$ What I dont understand here is that $p(y|\theta)$ is a probability distribution, i.e., y and $\theta$ are both random variables. However, in$ L(\theta|y)$ there is actually only one random variable, y, which is even kept fixed. And $\theta$ is treated as a normal parameter (i.e., not a random variable). So how can these functions be possibly proportional, since they might share the same mathematical formula, but they have completely different mathematical properties in the way how the variables are treated (random vs nonrandom)? $\endgroup$ – guest1 Feb 11 '20 at 19:21
  • $\begingroup$ They are proportional only with respect to the parameter $\theta$ (so this implicitly requires you to consider the density as a function of the parameters as well as the observable value). $\endgroup$ – Ben Feb 11 '20 at 21:28

Not the answer you're looking for? Browse other questions tagged or ask your own question.