0
$\begingroup$

I am refering to this course on sampling https://www.youtube.com/watch?v=TNZk8lo4e-Q. At around minute 6, the lecturer shows on the slide the posterior probability factored as:

$$p(\Theta |X,Y)=\frac{p(Y|X,\Theta)p(\Theta))}{Z}$$ where Z is the normalizing constant.

According to the product rule, the numerator should be

$$p(\Theta )p(X|\Theta )p(Y|X,\Theta )$$

What is the reason for dropping $p(X|\Theta )$? Thanks alot for your help!

$\endgroup$
6
  • 2
    $\begingroup$ Presumably the assumption is that the independent variable $X$ is independent of $\Theta$ (and may even have been decided by the experimenter) so $p(X\mid\Theta)$ is constant $p(X)$ for any particular $X$ and can be cancelled with the same constant in $Z$ $\endgroup$
    – Henry
    Jul 26, 2021 at 15:41
  • $\begingroup$ if $\Theta$ is independent of X, that means that $p(X|\Theta )$=$p(X)$. This is the density/pmf of x which is not constant for any particular X. How did u come to the conclusion that this is a constant? $\endgroup$
    – woowz
    Jul 26, 2021 at 15:58
  • 2
    $\begingroup$ with some abuse of notation $p(\Theta\mid X,Y)$ $=\dfrac{p(\Theta )p(X\mid \Theta )p(Y\mid X,\Theta )}{\int p(\Theta )p(X \mid \Theta)p(Y\mid X,\Theta )\, d\Theta} $ $=\dfrac{p(\Theta )p(X )p(Y\mid X,\Theta )}{\int p(\Theta )p(X )p(Y\mid X,\Theta )\, d\Theta} $ $= \dfrac{p(\Theta )p(Y\mid X,\Theta )}{\int p(\Theta )p(Y\mid X,\Theta )\, d\Theta}$ because the $p(X \mid \Theta)=p(X)$ do not vary with $\Theta$ $\endgroup$
    – Henry
    Jul 26, 2021 at 16:11
  • $\begingroup$ thanks! that does make sense. Since p(x) does not depend on $\theta$ it can be pulled out of the integral. But why should $\theta$ be independent of X? Aren't we learning the values of $\theta$ from X and Y, so they should be dependent of one another? $\endgroup$
    – woowz
    Jul 26, 2021 at 16:23
  • 2
    $\begingroup$ It is not my example, but the idea seems to be that $y$ (which you observe) is affected by $x$ (which you either set or observe) and $\theta$ (which is unknown but for which you have a prior distribution of belief), and you then use the likelihood to update your prior distribution to a posterior distribution for $\Theta$ $\endgroup$
    – Henry
    Jul 26, 2021 at 16:34

2 Answers 2

1
$\begingroup$

Consider that you have a linear regression where $Y$ is your output $X$ is your input and $\theta$ is your coefficients.

Then the posterior distribution of $\theta$ can be expressed as

$$p(\theta|X,y)=\frac{p(\theta,X,Y)}{p(X,Y)} = \frac{p(Y|X,\theta)p(X,\theta)}{p(X,Y)}$$

however, if you see a directed acyclic graph of Bayesian linear regression you will see that $\theta$ affects directly $Y$ and is independent of $X$, so $p(X,\theta)=P(X)p(\theta)$.

$$p(\theta|X,Y)= \frac{p(Y|X,\theta)p(X)p(\theta)}{p(X,Y)}$$

and divide the denominator and numerator with $p(X)$ and you take

$$p(\theta|X,Y) =\frac{p(Y|X,\theta)p(\theta)}{p(Y|X)}$$

$\endgroup$
1
  • 1
    $\begingroup$ Thanks! I have accepted this as the answer. Intuitively we can also say that theta is independent of X, because thats just saying we are applying a constant prior for every dataset that we get. $\endgroup$
    – woowz
    Jul 26, 2021 at 17:22
0
$\begingroup$

In the video, the normalizing constant is given as $$Z=\int p(Y|X,\theta)p(\theta)d\theta$$

All makes sense when the original equation is

$$p(\theta|X,Y)=\frac{p(Y|X,\theta)p(\theta|X)}{Z}$$

and $p(\theta|X)=p(\theta)$ because it is a prior in MCMC context. This is also the case for the normalizing constant.

$\endgroup$
2
  • $\begingroup$ Thanks, but then we still have the same question from the product rule. The numerator should be $p(Y|X,\theta)p(\theta|x)p(x)$. where did the p(x) go? $\endgroup$
    – woowz
    Jul 26, 2021 at 16:10
  • 1
    $\begingroup$ No, X is given everywhere, like $c$ here: $$p(a|b,c)=p(b|a,c)p(a|c)$$ $\endgroup$
    – gunes
    Jul 26, 2021 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.