3
$\begingroup$

In the linear regression model, we have $$\mathbf{y} \mid \boldsymbol\theta \sim \mathcal{N}(\mathbf{X}\boldsymbol\theta, \boldsymbol\Sigma_{\boldsymbol\eta})$$ $$\boldsymbol\theta \sim \mathcal{N}(\boldsymbol\theta_0, \boldsymbol\Sigma_0)$$

I am interested in finding the distribution of $\boldsymbol\theta \mid\mathbf{y}$.

From my work after Math StackExchange as well as a more general result that I used, as well as a more general result that I was given here in Stats SE, I obtain $$\boldsymbol\theta \mid \mathbf{y} \sim \mathcal{N}\left(\left(\mathbf{X}^{\prime}\boldsymbol\Sigma^{-1}_{\boldsymbol\eta}\mathbf{X}+\boldsymbol\Sigma^{-1}_0 \right)^{-1}(\boldsymbol\Sigma^{-1}_0\boldsymbol\theta_0+\mathbf{X}^{\prime}\boldsymbol\Sigma_{\boldsymbol\eta}^{-1}\mathbf{y}), \left(\mathbf{X}^{\prime}\boldsymbol\Sigma^{-1}_{\boldsymbol\eta}\mathbf{X}+\boldsymbol\Sigma^{-1}_0 \right)^{-1}\right)\text{.}\tag{1}$$ However, Theodoridis states that on p. 88 of Machine Learning that $$\mathbb{E}[\boldsymbol\theta \mid \mathbf{y}]=\boldsymbol\theta_0 + \left(\mathbf{X}^{\prime}\boldsymbol\Sigma^{-1}_{\boldsymbol\eta}\mathbf{X}+\boldsymbol\Sigma^{-1}_0 \right)^{-1}\mathbf{X}^{\prime}\boldsymbol\Sigma_{\boldsymbol\eta}^{-1}(\mathbf{y}-\mathbf{X}\boldsymbol\theta_0)\tag{2}$$ (our variance matrices agree).

I don't see how $\mathbb{E}[\boldsymbol\theta \mid \mathbf{y}]$ in $(1)$ and $(2)$ should agree. I attempted using the Woodbury identity and obtained $$\left(\mathbf{X}^{\prime}\boldsymbol\Sigma^{-1}_{\boldsymbol\eta}\mathbf{X}+\boldsymbol\Sigma^{-1}_0 \right)^{-1}=\boldsymbol\Sigma_0-\boldsymbol\Sigma_0\mathbf{X}^{\prime}(\boldsymbol\Sigma_{\boldsymbol\eta}+\mathbf{X}\boldsymbol\Sigma_0\mathbf{X}^{\prime})^{-1}\mathbf{X}\boldsymbol\Sigma_0$$ and when doing the multiplication as in $(1)$, I obtain $$\left(\mathbf{X}^{\prime}\boldsymbol\Sigma^{-1}_{\boldsymbol\eta}\mathbf{X}+\boldsymbol\Sigma^{-1}_0 \right)^{-1}(\boldsymbol\Sigma^{-1}_0\boldsymbol\theta_0+\mathbf{X}^{\prime}\boldsymbol\Sigma_{\boldsymbol\eta}^{-1}\mathbf{y})=\boldsymbol\theta_0+\boldsymbol\theta_0\mathbf{X}^{\prime}\boldsymbol\Sigma^{-1}_{\boldsymbol\eta}\mathbf{y}-\text{a mess...}$$ which doesn't look correct. Is there something I'm missing here?

$\endgroup$
2
  • 2
    $\begingroup$ In the expectation of (1) write $\boldsymbol{\Sigma}_0^{-1} \boldsymbol{\theta}_0$ as $[\boldsymbol{\Sigma}_0^{-1} + \mathbf{X}'\boldsymbol{\Sigma}_{\boldsymbol{\eta}}^{-1}\mathbf{X}] \boldsymbol{\theta}_0 - \mathbf{X}'\boldsymbol{\Sigma}_{\boldsymbol{\eta}}^{-1}\mathbf{X} \boldsymbol{\theta}_0 $. $\endgroup$
    – Yves
    Jan 7, 2018 at 15:04
  • 1
    $\begingroup$ @Yves Thank you very much! If you would like to put that as an answer, please do so, and I can award you points. $\endgroup$ Jan 7, 2018 at 16:15

1 Answer 1

2
$\begingroup$

In the expectation of (1) write $\boldsymbol{\Sigma}_0^{-1} \boldsymbol{\theta}_0$ as $[\boldsymbol{\Sigma}_0^{-1} + \mathbf{X}'\boldsymbol{\Sigma}_{\boldsymbol{\eta}}^{-1} \mathbf{X}] \, \boldsymbol{\theta}_0 - \mathbf{X}'\boldsymbol{\Sigma}_{\boldsymbol{\eta}}^{-1}\mathbf{X} \boldsymbol{\theta}_0 $. By simply rearranging we get (2).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.