0
$\begingroup$

In one of the slides from my class related to bayesian linear regression, I have the following scenario.

Under g-prior, the shrinkage estimator induced by the prior is $$\hat{\beta_{\alpha}} = \alpha\hat{\beta}$$

now it states that the value of $\alpha$ that minimizes the mean squared error (conditioned on $\beta$ and $\sigma^2$) for this estimator is

$$\alpha^* = \frac{1}{\frac{\sigma^2tr((X'X)^{-1})}{\beta'\beta}+1}$$

It also says that the proof uses the result: if $Z$ is a random vector with mean $\mu$ and covariance $\Sigma$ then with matrix $\Delta$ $$E[Z'\Delta Z] = tr(\Delta\Sigma) + \mu'\Delta\mu$$

Could anyone help deriving this $\alpha^*$? I am approaching by trying to differentiate $E[\| Y - X\hat{\beta_{\alpha}}\|^2 | \beta, \sigma^2]$ not really going anywhere.

$\endgroup$
2
$\begingroup$

It comes as \begin{align*} \mathbb{E}[\| Y - X\hat{\beta_{\alpha}}\|^2]&= \mathbb{E}[\| Y - X\beta\|^2] \overbrace{+}^{\text{Pythagore's}} \mathbb{E}[\| X\beta - X\hat{\beta_{\alpha}}\|^2]\\ &= \underbrace{C}_\text{independent of $\alpha$} + \mathbb{E}[(\beta - \hat{\beta_{\alpha}})'X'X (\beta - \hat{\beta_{\alpha}})]\\ &= C + \mathbb{E}[(\beta - \alpha\hat{\beta})'X'X (\hat\beta - \hat{\beta})]\\ &= C + \alpha^2\sigma^2\text{tr}\{X'X(X'X)^{-1}\}+(\beta - \alpha{\beta})'X'X (\beta - \alpha\beta)\\ &= C + \alpha^2\sigma^2\,p+(\alpha-1)^2{\beta}'X'X \beta \end{align*} Hence there seems to be a mistake in the formula as provided.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.