# Posterior predictive distribution(Bayes regression)

Assume we have some linear regression of the form $$y^{(i)}=\theta^{T}x^{(i)}+e^{(i)}$$

where: $$P(y^{(i)}|x^{(i)},\theta)=\frac{1}{\sqrt{2\pi}\sigma}exp(-\frac{(y^{(i)}-\theta^{T}x^{(i)})^{2}}{2\sigma^{2}})$$

Then using Bayes Rule we obtain some parameter distribution $p(\theta|S)$ where S is our dataset.

My problem is with derivation of posterior predictive distribution which is given in notes as follows:

Given new test point $x_{*}$ probability distribution over possible outputs(posterior predictive distribution) is:

$$p(y_{*}|x_{*},S)=\int_{\theta}p(y_{*}|x_{*},\theta)p(\theta|S)d\theta$$

And I dont fully understand why exactly we have $p(\theta|S)$ instead of $p(\theta)$ in above formula, because my reasoning is following:

$$p(y_{*}|x_{*},S)=\int_{\theta}p(y_{*}|x_{*},\theta,S)p(\theta)d\theta = \int_{\theta}p(y_{*}|x_{*},\theta)p(\theta)d\theta$$ (By marginal probability and independence of $x_{*}$ from $S$)

You have to condition on $S$ everywhere including in $\theta$ to take into account the information contained in $S$. By the marginalisation argument: $$p(y_*|x_*,S)=\int p(y_*,\theta|x_*,S)\text{d}\theta=\int p(y_*|x_*,S,\theta)p(\theta|S,x_{*})\text{d}\theta=\int p(y_*|x_*,S,\theta)p(\theta|S)\text{d}\theta$$
• I have still one question: why $p(y_{*}|x_{*},S,\theta)=p(y_{*}|x_{*}$,\theta)? cs229.stanford.edu/section/cs229-gaussian_processes.pdf page 4. On the one hand the formula (3) on page 4 suggest that $p(y_{*}|x_{*},\theta)$ is conditionally independent from S, on the other hand in the last formula on page 4 we see that distribution of $y_{*}$ is determined by $X$. You included S in final formula($p(y_{*}|x_{*},S,\theta)$. So what is the final answer: $p(y_*|x_*,S)=\int p(y_*|x_*,S,\theta)p(\theta|S)\text{d}\theta$ or $p(y_*|x_*,S)=\int p(y_*|x_*,\theta)p(\theta|S)\text{d}\theta$ – mokebe Oct 31 '16 at 20:21
• In some settings like iid observations, $y_*$ is independent of the sample $S$ given the parameter $\theta$ and $x_*$. In others like time-series, there may be dependence on the current sample even given $\theta$ if you predict the next observation. (I did not include $S$ in the conditioning, mokebe did. In doubt, you must keep the conditioning on $S$. – Xi'an Oct 31 '16 at 20:29