# Law of total probability on conditional

I often saw a formula, used mostly to integrate on the parameter space like:

$$p(x|y) = \int p(x|\theta) p(\theta|y) d\theta$$ where $\theta$ is the parameter.

I am confused and I hope to explain me how is that.

I know that $p(x) = \int p(x,\theta)d\theta = \int p(x|\theta)p(\theta)d\theta$. If I use a similar idea which in my mind means "a marginal distribution is obtain by integrating a join distribution over one variable", but with a conditional, I would expect to have:

$$p(x|y) = \int p(x,\theta|y)d\theta = \int p(x|\theta,y)p(\theta|y)d\theta$$ which obviously not not look the same.

Any help with that?

Later edit: The first formula was not correct, the integration is over $\theta$. Thanks

• Where did you see the formula? It's not correct. (There cannot be $y$ on the left hand side while $y$ is integrated over on the rhs). Jun 24 '14 at 11:19
• I study from Kevin Murphy - Machine Learning: A probabilistic perspective. There are multiple places. Section 3.5.2 Using model for prediction where is the formula used to translate 3.63 into 3.64.3.65; Section 3.4.4 Posterior predictive in formula 3.49; Section 3.3.4 Posterior predictive distribution in formula 3.28, to name just a few Jun 24 '14 at 13:21

Your derivation is correct. Now, if $x$ and $y$ are conditionally independent given $\theta$ you can do a last step in your derivation using $p(x\mid\theta,y)=p(x\mid\theta)$ and find $$p(x\mid y) = \int p(x\mid\theta)\,p(\theta\mid y)\,d\theta .$$ The intuitive idea is that, when $x$ and $y$ are conditionally independent given $\theta$, if you now the value of $\theta$ then you know the distribution of $x$, and you can "discard" the information about $x$ given by $y$ in $p(x\mid\theta,y)$. Notice that the information about $x$ contained in $y$ just updates your prior from $p(\theta)$ to the posterior $p(\theta\mid y)$ (compare with $p(x)=\int p(x\mid\theta)\,p(\theta)\,d\theta$). In the non cIID case you will have exactly your formula.
• So, the formula is valid only if $(x \perp y) \mid \theta$ ($x$ and $y$ are independent given $\theta$), right? If it not the case what I have is the right formula. I will search for clues about that conditional independence. Thank you. Jun 24 '14 at 13:25