How to derive prediction equations for classification in a bayesian setting? I'm following Bayesian Methods for Machine Learning course on Coursera and the following equations are given for training and prediction without derivation:
Training:
$$ P(\theta | X_{tr}, y_{tr}) = \frac{P(y_{tr} | X_{tr}, \theta), P(\theta)}{P(y_{tr} | X_{tr})}$$
and Prediction:
$$ P(y_{ts} | X_{ts}, X_{tr}, y_{tr}) = \int P(y_{ts} | X_{ts}, \theta) P(\theta | X_{tr}, y_{tr}) d \theta $$
where $ts$ refers to test data, and $tr$ to training.
I was able to derive the expression for training but I'm stuck on the prediction equation
References
Bayesian Methods for Machine Learning, Coursera, Week 1, "Bayesian approach to statistics".
 A: The marginalization rule of probability for a joint distribution $P(X,\theta)$ is
$$ P(X) = \int P(X,\theta) d\theta$$
on the other hand, the product rule says that $P(X,\theta) = P(X|\theta)P(\theta)$. Combining both
$$ P(X) = \int P(X|\theta)P(\theta) d\theta$$
In the case of having training that, you need to add conditions to the distributions, but nothing changes fundamentally. For notational simplicity, let's call $X$ and $Y$ the training data, $X'$ the point at which you want the prediction and $Y'$ the predicted value. You have a parameter $\theta$. You look for $P(Y'|X',X,Y)$. To find this, we apply the marginalization rule for the parameter $\theta$, thus
$$ P(Y'|X',X,Y) = \int P(Y'|X',X,Y,\theta)P(\theta|X',X,Y) d\theta$$
The last step is to realize that some of the conditions you have are unnecessary. For predicting $Y'$ you have a model that needs only $X'$ and $\theta$, but not the training data $X$,$Y$ explicitly (because the training data is used only for finding $\theta$). Thus, once you have $\theta$, $P(Y'|X',X,Y,\theta) \to P(Y'|X',\theta)$. On the other hand, how you obtain $\theta$ depends only on your training data, and obviously not on which value $X'$ you will want to make a future prediction, thus $P(\theta|X',X,Y) \to P(\theta|X,Y)$. The final expression is thus
$$ P(Y'|X',X,Y) = \int P(Y'|X',\theta)P(\theta|X,Y) d\theta$$
which is what you were looking for except for the change in the notation.
