0
$\begingroup$

I was reading the book Gaussian Processes for Machine Learning by C. E. Rasmussen & C. K. I. Williams. In the Regression chapter (chapter 2) they teach you how to do a linear regression for data $X \in \mathbb{R}^{n \times d}$ and $y \in\mathbb{R}^{n \times 1}$ in the sense of maximum likelihood with a gaussian prior on the weights and gaussian form assumed for data :

$p(y|w,X) \propto \exp(-\frac{1}{2\sigma_{n}^{2}}(y-Xw)^{T}(y-Xw))$

$p(w) \propto \exp(-\frac{1}{2}w^{T}\Sigma_{p}^{-1}w)$

So by direct application of Bayes rule you have :

$p(w|y,X) \propto p(y|w,X)p(w)$

$p(x|y,X)\propto \exp(-\frac{1}{2\sigma_{n}^{2}}y^{T}y+\frac{1}{2\sigma_{n}^{2}}y^{T}Xw+\frac{1}{2\sigma_{n}^{2}}w^{T}X^{T}y-\frac{1}{2\sigma_{n}^{2}}w^{T}X^{T}Xw-\frac{1}{2}w^{T}\Sigma_{p}^{-1}w)$

And then by some obscure magic you get:

$p(x|Y,X)\propto \exp(-\frac{1}{2}(w-\bar{w})^{T}(\frac{1}{\sigma_{n}^{2}}XX^{T}+\Sigma_{p}^{-1})(w-\bar{w}))$

with $\bar{w}=\sigma_{n}^{-2}(\sigma_{n}^{-2}XX^{T}+\Sigma_{p}^{-1})^{-1}Xy$.

I read somewhere it could come from the Woodbury formula indeed the final $\bar{w}$ is of the form $(A+UCV)^{-1}$ but I just cannot figure it out. There must be some simple trick to make sens out of it.

EDIT 1: I got to the result in the case where there is no $\Sigma_{p}$ but I am still at a loss regarding the general case I did something like:

$\forall W \in \mathbb{R}^{d \times 1}:$

$(y-Xw)^{T}(y-Xw)=(y-XW+XW-Xw)^{T}(y-XW+XW-Xw)=(y-XW)^{T}(y-XW)+(y-XW)^{T}(X)(W-w)+(W-w)^{T}X^{T}(y-XW)+(W-w)^{T}X^{T}X(W-w)$

As they are all scalar I can transpose the second term which gives:

$=||y-XW||_{2}^{2}+2(y-XW)^{T}X(W-w)+(w-W)^{T}X^{T}X(w-W)$

My equality still holds true for $W=\bar{w}$.

Then we do not care about the first term of the equality as it is constant w.r.t w, the second term evaluates to 0 if there is no $\Sigma_{p}$ but in the general case I end up with: $p(w|X,y) \propto \exp(-\frac{1}{2\sigma_{n}^{2}}\left(-2(y-X\bar{w})^{T}X(w-\bar{w})+(w-\bar{w})^{T}X^{T}X(w-\bar{w})\right)-\frac{1}{2}(w^{T}\Sigma_{p}^{-1}w))$
And I cannot manage to simplify it...

$\endgroup$

1 Answer 1

0
$\begingroup$

The answer I found is very complicated I do not know if there is a direct method (?). The expression above for the likelihood*prior can be "easily"(or at least in a more simple way) simplified using vector z=$(\beta, y)$:

$p(w|X,y)=\exp{z^{T}Rz}$

Then once you got there you use woodbury inversion formula to get the covariance of z. Then you use Theorem 4 part b to extract the covariance and mean of conditional $\beta|y$.

The process can be found in Bishop's Pattern and Recognition for ML Book on chapter 2 section 2.3.3..

I have no idea why my question was downvoted.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.