# How to derive prediction equations for classification in a bayesian setting?

I'm following Bayesian Methods for Machine Learning course on Coursera and the following equations are given for training and prediction without derivation:

Training:

$$P(\theta | X_{tr}, y_{tr}) = \frac{P(y_{tr} | X_{tr}, \theta), P(\theta)}{P(y_{tr} | X_{tr})}$$

and Prediction:

$$P(y_{ts} | X_{ts}, X_{tr}, y_{tr}) = \int P(y_{ts} | X_{ts}, \theta) P(\theta | X_{tr}, y_{tr}) d \theta$$

where $$ts$$ refers to test data, and $$tr$$ to training.

I was able to derive the expression for training but I'm stuck on the prediction equation

References

Bayesian Methods for Machine Learning, Coursera, Week 1, "Bayesian approach to statistics".

The marginalization rule of probability for a joint distribution $$P(X,\theta)$$ is

$$P(X) = \int P(X,\theta) d\theta$$

on the other hand, the product rule says that $$P(X,\theta) = P(X|\theta)P(\theta)$$. Combining both

$$P(X) = \int P(X|\theta)P(\theta) d\theta$$

In the case of having training that, you need to add conditions to the distributions, but nothing changes fundamentally. For notational simplicity, let's call $$X$$ and $$Y$$ the training data, $$X'$$ the point at which you want the prediction and $$Y'$$ the predicted value. You have a parameter $$\theta$$. You look for $$P(Y'|X',X,Y)$$. To find this, we apply the marginalization rule for the parameter $$\theta$$, thus

$$P(Y'|X',X,Y) = \int P(Y'|X',X,Y,\theta)P(\theta|X',X,Y) d\theta$$

The last step is to realize that some of the conditions you have are unnecessary. For predicting $$Y'$$ you have a model that needs only $$X'$$ and $$\theta$$, but not the training data $$X$$,$$Y$$ explicitly (because the training data is used only for finding $$\theta$$). Thus, once you have $$\theta$$, $$P(Y'|X',X,Y,\theta) \to P(Y'|X',\theta)$$. On the other hand, how you obtain $$\theta$$ depends only on your training data, and obviously not on which value $$X'$$ you will want to make a future prediction, thus $$P(\theta|X',X,Y) \to P(\theta|X,Y)$$. The final expression is thus

$$P(Y'|X',X,Y) = \int P(Y'|X',\theta)P(\theta|X,Y) d\theta$$

which is what you were looking for except for the change in the notation.