Assume we have some linear regression of the form $$y^{(i)}=\theta^{T}x^{(i)}+e^{(i)}$$

where: $$P(y^{(i)}|x^{(i)},\theta)=\frac{1}{\sqrt{2\pi}\sigma}exp(-\frac{(y^{(i)}-\theta^{T}x^{(i)})^{2}}{2\sigma^{2}})$$

Then using Bayes Rule we obtain some parameter distribution $p(\theta|S)$ where S is our dataset.

My problem is with derivation of posterior predictive distribution which is given in notes as follows:

Given new test point $x_{*}$ probability distribution over possible outputs(posterior predictive distribution) is:

$$p(y_{*}|x_{*},S)=\int_{\theta}p(y_{*}|x_{*},\theta)p(\theta|S)d\theta $$

And I dont fully understand why exactly we have $p(\theta|S)$ instead of $p(\theta)$ in above formula, because my reasoning is following:

$$p(y_{*}|x_{*},S)=\int_{\theta}p(y_{*}|x_{*},\theta,S)p(\theta)d\theta = \int_{\theta}p(y_{*}|x_{*},\theta)p(\theta)d\theta $$ (By marginal probability and independence of $x_{*}$ from $S$)


1 Answer 1


You have to condition on $S$ everywhere including in $\theta$ to take into account the information contained in $S$. By the marginalisation argument: $$p(y_*|x_*,S)=\int p(y_*,\theta|x_*,S)\text{d}\theta=\int p(y_*|x_*,S,\theta)p(\theta|S,x_{*})\text{d}\theta=\int p(y_*|x_*,S,\theta)p(\theta|S)\text{d}\theta$$

  • $\begingroup$ I have still one question: why $p(y_{*}|x_{*},S,\theta)=p(y_{*}|x_{*}$,\theta)? cs229.stanford.edu/section/cs229-gaussian_processes.pdf page 4. On the one hand the formula (3) on page 4 suggest that $p(y_{*}|x_{*},\theta)$ is conditionally independent from S, on the other hand in the last formula on page 4 we see that distribution of $y_{*}$ is determined by $X$. You included S in final formula($p(y_{*}|x_{*},S,\theta)$. So what is the final answer: $p(y_*|x_*,S)=\int p(y_*|x_*,S,\theta)p(\theta|S)\text{d}\theta$ or $p(y_*|x_*,S)=\int p(y_*|x_*,\theta)p(\theta|S)\text{d}\theta$ $\endgroup$
    – mokebe
    Commented Oct 31, 2016 at 20:21
  • $\begingroup$ In some settings like iid observations, $y_*$ is independent of the sample $S$ given the parameter $\theta$ and $x_*$. In others like time-series, there may be dependence on the current sample even given $\theta$ if you predict the next observation. (I did not include $S$ in the conditioning, mokebe did. In doubt, you must keep the conditioning on $S$. $\endgroup$
    – Xi'an
    Commented Oct 31, 2016 at 20:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.