What is the predictive distribution of Bayesian supervised Learning? (rigorous argument) I was trying to understand the posterior predictive distribution for any supervised predictor (by that I mean any classifier or regression predictor $f$). The exact equation I am unsure of is:
$$ p(y_{test} | x_{test} , S ) = \int_{\theta} p( y_{test} | x_{test} , \theta) p( \theta | S) d\theta $$
The exact source of confusion I have is how the above equation was derived from the conditional independence assumptions. 
These are my thoughts: To figure this out I tried drawing a graphical model for the above model but was unsure of what the correct model looked exactly. Usually, we assume that the data is generated from some true distribution $(x,y)  \sim P^*(x,y)$. If that is the case then I decided to draw the following graphical model as follows:

So according to my model:
$$(x^{(i)} , y^{(i)}) \perp (x^{(j)} , y^{(j)}) \mid \theta$$
if we consider what $p(y_{test} | x_{test} , S )$ really means via marginalization we get:
$$ p(y_{test} | x_{test} , S ) = \int_{\theta} p( y_{test}, \theta | x_{test} , S) d\theta  = \int_{\theta} p( y_{test} | \theta, x_{test} , S) p( \theta | S) d\theta $$
So how do we get the last step to be equal to:
$$ \int_{\theta} p( y_{test} | x_{test} , \theta) p( \theta | S) d\theta $$
why can we cross off/ignore the data  $S = \{ (x^{(i)}, y^{(i)})^n_{i = 1}\}$ given $\theta$?
 A: While doing the question I realized that the question basically boiled down to the following implication:
$$(x^{(i)} , y^{(i)}) \perp (x^{(j)} , y^{(j)}) \mid \theta \implies y^{(i)} \perp (x^{(j)} , y^{(j)}) \mid \theta$$
Or more precisely:
$$(x^{(i)} , y^{(i)}) \perp \{ (x^{(j)} , y^{(j)}) \}^n_{i=1} \mid \theta \implies y^{(i)} \perp  \{ (x^{(j)} , y^{(j)}) \}^n_{i=1} \mid \theta$$
$$(x^{(i)} , y^{(i)}) \perp S \mid \theta \implies y^{(i)} \perp  S \mid \theta$$
Intuitively, if x and y are conditionally independent of any other x and y, is y independent of a pair x and y? The answer is obviously yes.
I will prove the first implication, but the second one is easy to extend:
$(x^{(i)} , y^{(i)}) \perp (x^{(j)} , y^{(j)}) \mid \theta $ means the following:
$$P(x^{(i)} , y^{(i)}, x^{(j)} , y^{(j)} \mid \theta) = P(x^{(i)} , y^{(i)} \mid \theta) P( x^{(j)} , y^{(j)} \mid \theta )$$
and we want to show that given the above factorization that the following is true:
$$P(y^{(i)}, x^{(j)} , y^{(j)} \mid \theta) = P(y^{(i)} \mid \theta ) P( x^{(j)} , y^{(j)} \mid \theta)$$
Consider writing $P(y^{(i)}, x^{(j)} , y^{(j)})$ with its marginalization:
$$P(y^{(i)}, x^{(j)} , y^{(j)} \mid \theta) = \int_{x^{(i)}} P(x^{(i)} , y^{(i)}, x^{(j)} , y^{(j)} \mid \theta) dx = \int_{x^{(i)}} P(x^{(i)} , y^{(i)} \mid \theta ) P(x^{(j)} , y^{(j)} \mid \theta ) dx$$
then, now that we have it on the above form, we can factor out the things irrelevant to the summation:
$$P(y^{(i)}, x^{(j)} , y^{(j)} \mid \theta ) = P( x^{(j)} , y^{(j)} \mid \theta ) \int_{x^{(i)}} P(x^{(i)} , y^{(i)} \mid \theta ) dx = P(y^{(i)} \mid \theta ) P( x^{(j)} , y^{(j)} \mid \theta ) $$
and hence, get the desired factorization. Hence, we can factor the Bayesian predictive model as desired.

For a summary of the extended version:
$$P(y^{(i)}, S \mid \theta) = \int_{x^{(i)}} P(x^{(i)} , y^{(i)}, S \mid \theta) dx = \int_{x^{(i)}} P(x^{(i)} , y^{(i)} \mid \theta ) P( S \mid \theta ) dx$$
$$P(y^{(i)} , S \mid \theta ) = P( S \mid \theta ) \int_{x^{(i)}} P(x^{(i)} , y^{(i)} \mid \theta ) dx = P( S \mid \theta ) P(y^{(i)} \mid \theta ) $$

Last caveat, we actually need to show:
$$y^{(i)} \perp  S \mid (\theta, x^{(i)} )$$
This last thing can be addressed by noticing that what we are trying to show is that the label is independent of all the rest of the data given $\theta$. Obviously, that still holds but $x^{(i)}$ clearly affects the value of $y^{(i)}$, even if its independent of the rest of the data.
Lets actually show it formally.
I want to show that:
$$ x,y \perp S \mid \theta \implies p(y, S | \theta, x) = p(y \mid \theta, x) p(S \mid \theta)$$
So lets write out what $p(y, S | \theta, x)$ is:
$$ p(y, S | \theta, x) = \frac{p(x,y, S , \theta) }{p(x, \theta)} = \frac{p(x,y, S| \theta) p(\theta)}{p(\theta, x)} = \frac{p(x,y | \theta)p(S | \theta) p(\theta)}{p(\theta, x)} $$
$$ \frac{p(x, y | \theta) p(\theta)}{p(\theta, x)} p(S |\theta) = \frac{p(x, y, \theta)}{p(\theta, x)} p(S |\theta) = p(y \mid \theta, x) p(S \mid \theta) $$
So we finally have what we truly needed.
