I often saw a formula, used mostly to integrate on the parameter space like:

$$ p(x|y) = \int p(x|\theta) p(\theta|y) d\theta $$ where $\theta$ is the parameter.

I am confused and I hope to explain me how is that.

I know that $p(x) = \int p(x,\theta)d\theta = \int p(x|\theta)p(\theta)d\theta $. If I use a similar idea which in my mind means "a marginal distribution is obtain by integrating a join distribution over one variable", but with a conditional, I would expect to have:

$$ p(x|y) = \int p(x,\theta|y)d\theta = \int p(x|\theta,y)p(\theta|y)d\theta$$ which obviously not not look the same.

Any help with that?

Later edit: The first formula was not correct, the integration is over $\theta$. Thanks

  • 1
    $\begingroup$ Where did you see the formula? It's not correct. (There cannot be $y$ on the left hand side while $y$ is integrated over on the rhs). $\endgroup$ Jun 24 '14 at 11:19
  • $\begingroup$ I study from Kevin Murphy - Machine Learning: A probabilistic perspective. There are multiple places. Section 3.5.2 Using model for prediction where is the formula used to translate 3.63 into; Section 3.4.4 Posterior predictive in formula 3.49; Section 3.3.4 Posterior predictive distribution in formula 3.28, to name just a few $\endgroup$
    – rapaio
    Jun 24 '14 at 13:21

Your derivation is correct. Now, if $x$ and $y$ are conditionally independent given $\theta$ you can do a last step in your derivation using $p(x\mid\theta,y)=p(x\mid\theta)$ and find $$ p(x\mid y) = \int p(x\mid\theta)\,p(\theta\mid y)\,d\theta . $$ The intuitive idea is that, when $x$ and $y$ are conditionally independent given $\theta$, if you now the value of $\theta$ then you know the distribution of $x$, and you can "discard" the information about $x$ given by $y$ in $p(x\mid\theta,y)$. Notice that the information about $x$ contained in $y$ just updates your prior from $p(\theta)$ to the posterior $p(\theta\mid y)$ (compare with $p(x)=\int p(x\mid\theta)\,p(\theta)\,d\theta$). In the non cIID case you will have exactly your formula.

  • $\begingroup$ So, the formula is valid only if $ (x \perp y) \mid \theta $ ($x$ and $y$ are independent given $\theta$), right? If it not the case what I have is the right formula. I will search for clues about that conditional independence. Thank you. $\endgroup$
    – rapaio
    Jun 24 '14 at 13:25
  • $\begingroup$ Dawid provides amazing intuitions on the subject: edlab-www.cs.umass.edu/cs589/2010-lectures/… $\endgroup$
    – Zen
    Jun 24 '14 at 15:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.