I'm following Bayesian Methods for Machine Learning course on Coursera and the following equations are given for training and prediction without derivation:


$$ P(\theta | X_{tr}, y_{tr}) = \frac{P(y_{tr} | X_{tr}, \theta), P(\theta)}{P(y_{tr} | X_{tr})}$$

and Prediction:

$$ P(y_{ts} | X_{ts}, X_{tr}, y_{tr}) = \int P(y_{ts} | X_{ts}, \theta) P(\theta | X_{tr}, y_{tr}) d \theta $$

where $ts$ refers to test data, and $tr$ to training.

I was able to derive the expression for training but I'm stuck on the prediction equation


Bayesian Methods for Machine Learning, Coursera, Week 1, "Bayesian approach to statistics".


1 Answer 1


The marginalization rule of probability for a joint distribution $P(X,\theta)$ is

$$ P(X) = \int P(X,\theta) d\theta$$

on the other hand, the product rule says that $P(X,\theta) = P(X|\theta)P(\theta)$. Combining both

$$ P(X) = \int P(X|\theta)P(\theta) d\theta$$

In the case of having training that, you need to add conditions to the distributions, but nothing changes fundamentally. For notational simplicity, let's call $X$ and $Y$ the training data, $X'$ the point at which you want the prediction and $Y'$ the predicted value. You have a parameter $\theta$. You look for $P(Y'|X',X,Y)$. To find this, we apply the marginalization rule for the parameter $\theta$, thus

$$ P(Y'|X',X,Y) = \int P(Y'|X',X,Y,\theta)P(\theta|X',X,Y) d\theta$$

The last step is to realize that some of the conditions you have are unnecessary. For predicting $Y'$ you have a model that needs only $X'$ and $\theta$, but not the training data $X$,$Y$ explicitly (because the training data is used only for finding $\theta$). Thus, once you have $\theta$, $P(Y'|X',X,Y,\theta) \to P(Y'|X',\theta)$. On the other hand, how you obtain $\theta$ depends only on your training data, and obviously not on which value $X'$ you will want to make a future prediction, thus $P(\theta|X',X,Y) \to P(\theta|X,Y)$. The final expression is thus

$$ P(Y'|X',X,Y) = \int P(Y'|X',\theta)P(\theta|X,Y) d\theta$$

which is what you were looking for except for the change in the notation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.