# What is the predictive distribution of Bayesian supervised Learning? (rigorous argument)

I was trying to understand the posterior predictive distribution for any supervised predictor (by that I mean any classifier or regression predictor $f$). The exact equation I am unsure of is:

$$p(y_{test} | x_{test} , S ) = \int_{\theta} p( y_{test} | x_{test} , \theta) p( \theta | S) d\theta$$

The exact source of confusion I have is how the above equation was derived from the conditional independence assumptions.

These are my thoughts: To figure this out I tried drawing a graphical model for the above model but was unsure of what the correct model looked exactly. Usually, we assume that the data is generated from some true distribution $(x,y) \sim P^*(x,y)$. If that is the case then I decided to draw the following graphical model as follows:

So according to my model:

$$(x^{(i)} , y^{(i)}) \perp (x^{(j)} , y^{(j)}) \mid \theta$$

if we consider what $p(y_{test} | x_{test} , S )$ really means via marginalization we get:

$$p(y_{test} | x_{test} , S ) = \int_{\theta} p( y_{test}, \theta | x_{test} , S) d\theta = \int_{\theta} p( y_{test} | \theta, x_{test} , S) p( \theta | S) d\theta$$

So how do we get the last step to be equal to:

$$\int_{\theta} p( y_{test} | x_{test} , \theta) p( \theta | S) d\theta$$

why can we cross off/ignore the data $S = \{ (x^{(i)}, y^{(i)})^n_{i = 1}\}$ given $\theta$?

While doing the question I realized that the question basically boiled down to the following implication:

$$(x^{(i)} , y^{(i)}) \perp (x^{(j)} , y^{(j)}) \mid \theta \implies y^{(i)} \perp (x^{(j)} , y^{(j)}) \mid \theta$$

Or more precisely:

$$(x^{(i)} , y^{(i)}) \perp \{ (x^{(j)} , y^{(j)}) \}^n_{i=1} \mid \theta \implies y^{(i)} \perp \{ (x^{(j)} , y^{(j)}) \}^n_{i=1} \mid \theta$$

$$(x^{(i)} , y^{(i)}) \perp S \mid \theta \implies y^{(i)} \perp S \mid \theta$$

Intuitively, if x and y are conditionally independent of any other x and y, is y independent of a pair x and y? The answer is obviously yes.

I will prove the first implication, but the second one is easy to extend:

$(x^{(i)} , y^{(i)}) \perp (x^{(j)} , y^{(j)}) \mid \theta$ means the following:

$$P(x^{(i)} , y^{(i)}, x^{(j)} , y^{(j)} \mid \theta) = P(x^{(i)} , y^{(i)} \mid \theta) P( x^{(j)} , y^{(j)} \mid \theta )$$

and we want to show that given the above factorization that the following is true:

$$P(y^{(i)}, x^{(j)} , y^{(j)} \mid \theta) = P(y^{(i)} \mid \theta ) P( x^{(j)} , y^{(j)} \mid \theta)$$

Consider writing $P(y^{(i)}, x^{(j)} , y^{(j)})$ with its marginalization:

$$P(y^{(i)}, x^{(j)} , y^{(j)} \mid \theta) = \int_{x^{(i)}} P(x^{(i)} , y^{(i)}, x^{(j)} , y^{(j)} \mid \theta) dx = \int_{x^{(i)}} P(x^{(i)} , y^{(i)} \mid \theta ) P(x^{(j)} , y^{(j)} \mid \theta ) dx$$

then, now that we have it on the above form, we can factor out the things irrelevant to the summation:

$$P(y^{(i)}, x^{(j)} , y^{(j)} \mid \theta ) = P( x^{(j)} , y^{(j)} \mid \theta ) \int_{x^{(i)}} P(x^{(i)} , y^{(i)} \mid \theta ) dx = P(y^{(i)} \mid \theta ) P( x^{(j)} , y^{(j)} \mid \theta )$$

and hence, get the desired factorization. Hence, we can factor the Bayesian predictive model as desired.

For a summary of the extended version:

$$P(y^{(i)}, S \mid \theta) = \int_{x^{(i)}} P(x^{(i)} , y^{(i)}, S \mid \theta) dx = \int_{x^{(i)}} P(x^{(i)} , y^{(i)} \mid \theta ) P( S \mid \theta ) dx$$

$$P(y^{(i)} , S \mid \theta ) = P( S \mid \theta ) \int_{x^{(i)}} P(x^{(i)} , y^{(i)} \mid \theta ) dx = P( S \mid \theta ) P(y^{(i)} \mid \theta )$$

Last caveat, we actually need to show:

$$y^{(i)} \perp S \mid (\theta, x^{(i)} )$$

This last thing can be addressed by noticing that what we are trying to show is that the label is independent of all the rest of the data given $\theta$. Obviously, that still holds but $x^{(i)}$ clearly affects the value of $y^{(i)}$, even if its independent of the rest of the data.

Lets actually show it formally.

I want to show that:

$$x,y \perp S \mid \theta \implies p(y, S | \theta, x) = p(y \mid \theta, x) p(S \mid \theta)$$

So lets write out what $p(y, S | \theta, x)$ is:

$$p(y, S | \theta, x) = \frac{p(x,y, S , \theta) }{p(x, \theta)} = \frac{p(x,y, S| \theta) p(\theta)}{p(\theta, x)} = \frac{p(x,y | \theta)p(S | \theta) p(\theta)}{p(\theta, x)}$$ $$\frac{p(x, y | \theta) p(\theta)}{p(\theta, x)} p(S |\theta) = \frac{p(x, y, \theta)}{p(\theta, x)} p(S |\theta) = p(y \mid \theta, x) p(S \mid \theta)$$

So we finally have what we truly needed.