Variance of the ridge regression estimator I have some concerns about the image below (note that $\mathbf W_{\lambda} = (\mathbf X^\top \mathbf X + \lambda \mathbf I)^{-1} \mathbf X^\top \mathbf X$):

My main concern is that this derivation of the variance of the ridge regression estimator makes the assumption that the regular least squares estimator $\hat{\boldsymbol{\beta}}$ exists. (In particular, the assumption that $(\mathbf X^\top \mathbf X)^{-1}$ exists.)
I see that the final expression doesn't rely on $(\mathbf X^\top \mathbf X)^{-1}$, but I'm struggling to convince myself that this is a legitimate derivation that holds in all circumstances. For example, in high-dimensional regression (i.e. $n < p$), this is invalid, right?
I appreciate any help.
 A: That's a legitimate concern.  But since $\hat\beta_\lambda$ is a linear combination of the response $y,$ the explanation ought to go back to $y,$ thus:
$$\hat\beta_\lambda = (X^\prime X + \lambda)^{-1} X^\prime y.$$
Recall that (conditional on $X$) the components of $y$ are independent (and therefore uncorrelated) variables with common variance $\sigma^2.$  If you like, in matrix form this assumption can be written
$$\operatorname{Var}(y) = \sigma^2 \mathbb{I}_{nn}.$$
Thus the variance of $\hat\beta_\lambda$ is the sum of these variances, equal to
$$\begin{aligned}
\operatorname{Var}(\hat\beta_\lambda) &= (X^\prime X + \lambda\mathbb{I}_{pp})^{-1} X^\prime\, \sigma^2 \mathbb{I}_{nn}\, \left[(X^\prime X + \lambda\mathbb{I}_{pp})^{-1} X^\prime\right]^\prime
\end{aligned}$$
and this readily simplifies to the expression in the question because (1) the $\sigma^2\mathbb{I}_{nn}$ factors out and (2) basic properties of the transpose imply $\left[(X^\prime X + \lambda\mathbb{I}_{pp})^{-1} X^\prime\right]^\prime = X \left[(X^\prime X + \lambda\mathbb{I}_{pp})^{-1}\right]^\prime.$
