2
$\begingroup$

Assume we have some linear regression of the form $$y^{(i)}=\theta^{T}x^{(i)}+e^{(i)}$$

where: $$P(y^{(i)}|x^{(i)},\theta)=\frac{1}{\sqrt{2\pi}\sigma}exp(-\frac{(y^{(i)}-\theta^{T}x^{(i)})^{2}}{2\sigma^{2}})$$

Then using Bayes Rule we obtain some parameter distribution $p(\theta|S)$ where S is our dataset.

My problem is with derivation of posterior predictive distribution which is given in notes as follows:

Given new test point $x_{*}$ probability distribution over possible outputs(posterior predictive distribution) is:

$$p(y_{*}|x_{*},S)=\int_{\theta}p(y_{*}|x_{*},\theta)p(\theta|S)d\theta $$

And I dont fully understand why exactly we have $p(\theta|S)$ instead of $p(\theta)$ in above formula, because my reasoning is following:

$$p(y_{*}|x_{*},S)=\int_{\theta}p(y_{*}|x_{*},\theta,S)p(\theta)d\theta = \int_{\theta}p(y_{*}|x_{*},\theta)p(\theta)d\theta $$ (By marginal probability and independence of $x_{*}$ from $S$)

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
1
$\begingroup$

You have to condition on $S$ everywhere including in $\theta$ to take into account the information contained in $S$. By the marginalisation argument: $$p(y_*|x_*,S)=\int p(y_*,\theta|x_*,S)\text{d}\theta=\int p(y_*|x_*,S,\theta)p(\theta|S,x_{*})\text{d}\theta=\int p(y_*|x_*,S,\theta)p(\theta|S)\text{d}\theta$$

Improve this answer
$\endgroup$
2
  • $\begingroup$ I have still one question: why $p(y_{*}|x_{*},S,\theta)=p(y_{*}|x_{*}$,\theta)? cs229.stanford.edu/section/cs229-gaussian_processes.pdf page 4. On the one hand the formula (3) on page 4 suggest that $p(y_{*}|x_{*},\theta)$ is conditionally independent from S, on the other hand in the last formula on page 4 we see that distribution of $y_{*}$ is determined by $X$. You included S in final formula($p(y_{*}|x_{*},S,\theta)$. So what is the final answer: $p(y_*|x_*,S)=\int p(y_*|x_*,S,\theta)p(\theta|S)\text{d}\theta$ or $p(y_*|x_*,S)=\int p(y_*|x_*,\theta)p(\theta|S)\text{d}\theta$ $\endgroup$
    – mokebe
    Oct 31, 2016 at 20:21
  • $\begingroup$ In some settings like iid observations, $y_*$ is independent of the sample $S$ given the parameter $\theta$ and $x_*$. In others like time-series, there may be dependence on the current sample even given $\theta$ if you predict the next observation. (I did not include $S$ in the conditioning, mokebe did. In doubt, you must keep the conditioning on $S$. $\endgroup$
    – Xi'an
    Oct 31, 2016 at 20:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.