2
$\begingroup$

In the linear regression model, we have $$\mathbf{y} \mid \boldsymbol\theta \sim \mathcal{N}(\mathbf{X}\boldsymbol\theta, \boldsymbol\Sigma_{\boldsymbol\eta})$$ $$\boldsymbol\theta \sim \mathcal{N}(\boldsymbol\theta_0, \boldsymbol\Sigma_0)$$

I am interested in finding the distribution of $\boldsymbol\theta \mid\mathbf{y}$.

From my work after Math StackExchange as well as a more general result that I used, as well as a more general result that I was given here in Stats SE, I obtain $$\boldsymbol\theta \mid \mathbf{y} \sim \mathcal{N}\left(\left(\mathbf{X}^{\prime}\boldsymbol\Sigma^{-1}_{\boldsymbol\eta}\mathbf{X}+\boldsymbol\Sigma^{-1}_0 \right)^{-1}(\boldsymbol\Sigma^{-1}_0\boldsymbol\theta_0+\mathbf{X}^{\prime}\boldsymbol\Sigma_{\boldsymbol\eta}^{-1}\mathbf{y}), \left(\mathbf{X}^{\prime}\boldsymbol\Sigma^{-1}_{\boldsymbol\eta}\mathbf{X}+\boldsymbol\Sigma^{-1}_0 \right)^{-1}\right)\text{.}\tag{1}$$ However, Theodoridis states that on p. 88 of Machine Learning that $$\mathbb{E}[\boldsymbol\theta \mid \mathbf{y}]=\boldsymbol\theta_0 + \left(\mathbf{X}^{\prime}\boldsymbol\Sigma^{-1}_{\boldsymbol\eta}\mathbf{X}+\boldsymbol\Sigma^{-1}_0 \right)^{-1}\mathbf{X}^{\prime}\boldsymbol\Sigma_{\boldsymbol\eta}^{-1}(\mathbf{y}-\mathbf{X}\boldsymbol\theta_0)\tag{2}$$ (our variance matrices agree).

I don't see how $\mathbb{E}[\boldsymbol\theta \mid \mathbf{y}]$ in $(1)$ and $(2)$ should agree. I attempted using the Woodbury identity and obtained $$\left(\mathbf{X}^{\prime}\boldsymbol\Sigma^{-1}_{\boldsymbol\eta}\mathbf{X}+\boldsymbol\Sigma^{-1}_0 \right)^{-1}=\boldsymbol\Sigma_0-\boldsymbol\Sigma_0\mathbf{X}^{\prime}(\boldsymbol\Sigma_{\boldsymbol\eta}+\mathbf{X}\boldsymbol\Sigma_0\mathbf{X}^{\prime})^{-1}\mathbf{X}\boldsymbol\Sigma_0$$ and when doing the multiplication as in $(1)$, I obtain $$\left(\mathbf{X}^{\prime}\boldsymbol\Sigma^{-1}_{\boldsymbol\eta}\mathbf{X}+\boldsymbol\Sigma^{-1}_0 \right)^{-1}(\boldsymbol\Sigma^{-1}_0\boldsymbol\theta_0+\mathbf{X}^{\prime}\boldsymbol\Sigma_{\boldsymbol\eta}^{-1}\mathbf{y})=\boldsymbol\theta_0+\boldsymbol\theta_0\mathbf{X}^{\prime}\boldsymbol\Sigma^{-1}_{\boldsymbol\eta}\mathbf{y}-\text{a mess...}$$ which doesn't look correct. Is there something I'm missing here?

$\endgroup$
  • 2
    $\begingroup$ In the expectation of (1) write $\boldsymbol{\Sigma}_0^{-1} \boldsymbol{\theta}_0$ as $[\boldsymbol{\Sigma}_0^{-1} + \mathbf{X}'\boldsymbol{\Sigma}_{\boldsymbol{\eta}}^{-1}\mathbf{X}] \boldsymbol{\theta}_0 - \mathbf{X}'\boldsymbol{\Sigma}_{\boldsymbol{\eta}}^{-1}\mathbf{X} \boldsymbol{\theta}_0 $. $\endgroup$ – Yves Jan 7 '18 at 15:04
  • 1
    $\begingroup$ @Yves Thank you very much! If you would like to put that as an answer, please do so, and I can award you points. $\endgroup$ – Clarinetist Jan 7 '18 at 16:15
2
$\begingroup$

In the expectation of (1) write $\boldsymbol{\Sigma}_0^{-1} \boldsymbol{\theta}_0$ as $[\boldsymbol{\Sigma}_0^{-1} + \mathbf{X}'\boldsymbol{\Sigma}_{\boldsymbol{\eta}}^{-1} \mathbf{X}] \, \boldsymbol{\theta}_0 - \mathbf{X}'\boldsymbol{\Sigma}_{\boldsymbol{\eta}}^{-1}\mathbf{X} \boldsymbol{\theta}_0 $. By simply rearranging we get (2).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.