I was trying to understand the posterior predictive distribution for any supervised predictor (by that I mean any classifier or regression predictor $f$). The exact equation I am unsure of is:

$$ p(y_{test} | x_{test} , S ) = \int_{\theta} p( y_{test} | x_{test} , \theta) p( \theta | S) d\theta $$

The exact source of confusion I have is how the above equation was derived from the conditional independence assumptions.

These are my thoughts: To figure this out I tried drawing a graphical model for the above model but was unsure of what the correct model looked exactly. Usually, we assume that the data is generated from some true distribution $(x,y) \sim P^*(x,y)$. If that is the case then I decided to draw the following graphical model as follows:

enter image description here

So according to my model:

$$(x^{(i)} , y^{(i)}) \perp (x^{(j)} , y^{(j)}) \mid \theta$$

if we consider what $p(y_{test} | x_{test} , S )$ really means via marginalization we get:

$$ p(y_{test} | x_{test} , S ) = \int_{\theta} p( y_{test}, \theta | x_{test} , S) d\theta = \int_{\theta} p( y_{test} | \theta, x_{test} , S) p( \theta | S) d\theta $$

So how do we get the last step to be equal to:

$$ \int_{\theta} p( y_{test} | x_{test} , \theta) p( \theta | S) d\theta $$

why can we cross off/ignore the data $S = \{ (x^{(i)}, y^{(i)})^n_{i = 1}\}$ given $\theta$?


1 Answer 1


While doing the question I realized that the question basically boiled down to the following implication:

$$(x^{(i)} , y^{(i)}) \perp (x^{(j)} , y^{(j)}) \mid \theta \implies y^{(i)} \perp (x^{(j)} , y^{(j)}) \mid \theta$$

Or more precisely:

$$(x^{(i)} , y^{(i)}) \perp \{ (x^{(j)} , y^{(j)}) \}^n_{i=1} \mid \theta \implies y^{(i)} \perp \{ (x^{(j)} , y^{(j)}) \}^n_{i=1} \mid \theta$$

$$(x^{(i)} , y^{(i)}) \perp S \mid \theta \implies y^{(i)} \perp S \mid \theta$$

Intuitively, if x and y are conditionally independent of any other x and y, is y independent of a pair x and y? The answer is obviously yes.

I will prove the first implication, but the second one is easy to extend:

$(x^{(i)} , y^{(i)}) \perp (x^{(j)} , y^{(j)}) \mid \theta $ means the following:

$$P(x^{(i)} , y^{(i)}, x^{(j)} , y^{(j)} \mid \theta) = P(x^{(i)} , y^{(i)} \mid \theta) P( x^{(j)} , y^{(j)} \mid \theta )$$

and we want to show that given the above factorization that the following is true:

$$P(y^{(i)}, x^{(j)} , y^{(j)} \mid \theta) = P(y^{(i)} \mid \theta ) P( x^{(j)} , y^{(j)} \mid \theta)$$

Consider writing $P(y^{(i)}, x^{(j)} , y^{(j)})$ with its marginalization:

$$P(y^{(i)}, x^{(j)} , y^{(j)} \mid \theta) = \int_{x^{(i)}} P(x^{(i)} , y^{(i)}, x^{(j)} , y^{(j)} \mid \theta) dx = \int_{x^{(i)}} P(x^{(i)} , y^{(i)} \mid \theta ) P(x^{(j)} , y^{(j)} \mid \theta ) dx$$

then, now that we have it on the above form, we can factor out the things irrelevant to the summation:

$$P(y^{(i)}, x^{(j)} , y^{(j)} \mid \theta ) = P( x^{(j)} , y^{(j)} \mid \theta ) \int_{x^{(i)}} P(x^{(i)} , y^{(i)} \mid \theta ) dx = P(y^{(i)} \mid \theta ) P( x^{(j)} , y^{(j)} \mid \theta ) $$

and hence, get the desired factorization. Hence, we can factor the Bayesian predictive model as desired.

For a summary of the extended version:

$$P(y^{(i)}, S \mid \theta) = \int_{x^{(i)}} P(x^{(i)} , y^{(i)}, S \mid \theta) dx = \int_{x^{(i)}} P(x^{(i)} , y^{(i)} \mid \theta ) P( S \mid \theta ) dx$$

$$P(y^{(i)} , S \mid \theta ) = P( S \mid \theta ) \int_{x^{(i)}} P(x^{(i)} , y^{(i)} \mid \theta ) dx = P( S \mid \theta ) P(y^{(i)} \mid \theta ) $$

Last caveat, we actually need to show:

$$y^{(i)} \perp S \mid (\theta, x^{(i)} )$$

This last thing can be addressed by noticing that what we are trying to show is that the label is independent of all the rest of the data given $\theta$. Obviously, that still holds but $x^{(i)}$ clearly affects the value of $y^{(i)}$, even if its independent of the rest of the data.

Lets actually show it formally.

I want to show that:

$$ x,y \perp S \mid \theta \implies p(y, S | \theta, x) = p(y \mid \theta, x) p(S \mid \theta)$$

So lets write out what $p(y, S | \theta, x)$ is:

$$ p(y, S | \theta, x) = \frac{p(x,y, S , \theta) }{p(x, \theta)} = \frac{p(x,y, S| \theta) p(\theta)}{p(\theta, x)} = \frac{p(x,y | \theta)p(S | \theta) p(\theta)}{p(\theta, x)} $$ $$ \frac{p(x, y | \theta) p(\theta)}{p(\theta, x)} p(S |\theta) = \frac{p(x, y, \theta)}{p(\theta, x)} p(S |\theta) = p(y \mid \theta, x) p(S \mid \theta) $$

So we finally have what we truly needed.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.