2
$\begingroup$

I have some concerns about the image below (note that $\mathbf W_{\lambda} = (\mathbf X^\top \mathbf X + \lambda \mathbf I)^{-1} \mathbf X^\top \mathbf X$):

enter image description here

My main concern is that this derivation of the variance of the ridge regression estimator makes the assumption that the regular least squares estimator $\hat{\boldsymbol{\beta}}$ exists. (In particular, the assumption that $(\mathbf X^\top \mathbf X)^{-1}$ exists.)

I see that the final expression doesn't rely on $(\mathbf X^\top \mathbf X)^{-1}$, but I'm struggling to convince myself that this is a legitimate derivation that holds in all circumstances. For example, in high-dimensional regression (i.e. $n < p$), this is invalid, right?

I appreciate any help.

Improve this question
$\endgroup$
5
  • $\begingroup$ The variance of $My$ for a random vector $y$ and a constant matrix $M$ is simply $M \textrm{var}(y) M^T$. Can you see what $M$ is in the case of ridge regression? $\endgroup$
    – user551504
    Commented Nov 15, 2021 at 0:03
  • $\begingroup$ @user551504 I'm not sure if you're referring to the response $\mathbf Y$ or you're just referring to any random vector. I understand the algebraic manipulations (I think), but it's the assumption that I don't understand. $\endgroup$
    – Novice
    Commented Nov 15, 2021 at 0:08
  • $\begingroup$ In the case of ridge regression, $M=(X^TX + \lambda I)^{-1} X^T$ and $y$ is a vector normal distribution. There are no assumptions to the variance decomposition that I described earlier. Based on this, can you see whether there's any assumptions needed? $\endgroup$
    – user551504
    Commented Nov 15, 2021 at 0:26
  • $\begingroup$ @user551504 I think I understand now. Thanks. So there are no assumptions needed about the existence of the least squares estimator. Seems like your way of deriving the result is much better than the one in the image, because it doesn't involve the least squares estimator and isn't any more difficult. $\endgroup$
    – Novice
    Commented Nov 15, 2021 at 0:42
  • $\begingroup$ Well, one assumption is that $y$ is normally distributed with covariance matrix $\sigma^2 I$. That's a pretty big assumption. By working with the distribution of $\hat\beta$ directly, its asymptotic distribution is what you described under more general conditions $\endgroup$
    – user551504
    Commented Nov 15, 2021 at 1:08

1 Answer 1

Reset to default
2
$\begingroup$

That's a legitimate concern. But since $\hat\beta_\lambda$ is a linear combination of the response $y,$ the explanation ought to go back to $y,$ thus:

$$\hat\beta_\lambda = (X^\prime X + \lambda)^{-1} X^\prime y.$$

Recall that (conditional on $X$) the components of $y$ are independent (and therefore uncorrelated) variables with common variance $\sigma^2.$ If you like, in matrix form this assumption can be written

$$\operatorname{Var}(y) = \sigma^2 \mathbb{I}_{nn}.$$

Thus the variance of $\hat\beta_\lambda$ is the sum of these variances, equal to

$$\begin{aligned} \operatorname{Var}(\hat\beta_\lambda) &= (X^\prime X + \lambda\mathbb{I}_{pp})^{-1} X^\prime\, \sigma^2 \mathbb{I}_{nn}\, \left[(X^\prime X + \lambda\mathbb{I}_{pp})^{-1} X^\prime\right]^\prime \end{aligned}$$

and this readily simplifies to the expression in the question because (1) the $\sigma^2\mathbb{I}_{nn}$ factors out and (2) basic properties of the transpose imply $\left[(X^\prime X + \lambda\mathbb{I}_{pp})^{-1} X^\prime\right]^\prime = X \left[(X^\prime X + \lambda\mathbb{I}_{pp})^{-1}\right]^\prime.$

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.