Given that the closed-form ridge regression solution is $\hat{\beta}_{ridge} = (X^TX+\lambda I)^{-1}X^TY$, show that ridge outputs correlations Given that the closed-form ridge regression solution is $\hat{\beta}_{ridge} = (X^TX+\lambda I)^{-1}X^TY$, show that ridge regression outputs are equal to the correlations used in correlation screening when $\lambda \rightarrow \infty$.
I'm not really sure how to approach this problem. I understand that as $\lambda \rightarrow \infty$, $\beta \rightarrow 0$, which implies that $Y = X \beta + \varepsilon$, so $Y = \varepsilon$. However, I feel like this is not the right track, since I believe $\varepsilon$ reflects the random variance from the individual data points and not the correlations. Any help/direction towards the solution is greatly appreciated!
 A: It is actually possible to obtain a general form for the variance matrix of the ridge-estimator, which holds for all penalty parameters $\lambda$.  This is not necessary for your present purposes, but I will do it anyway so that you can see a more general form of the present results.  To start with, the ridge-estimator can be written in terms of the underlying regression coefficient and error vector as:
$$\begin{align}
\hat{\boldsymbol{\beta}}(\lambda)
&= (\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} (\mathbf{x}^\text{T} \mathbf{Y}) \\[6pt]
&= (\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} \mathbf{x}^\text{T} (\mathbf{x} \boldsymbol{\beta} + \boldsymbol{\varepsilon}) \\[6pt]
&= (\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} (\mathbf{x}^\text{T} \mathbf{x}) \boldsymbol{\beta} + (\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} \mathbf{x}^\text{T} \boldsymbol{\varepsilon}. \\[6pt]
\end{align}$$
As $\lambda \rightarrow \infty$ we get the asymptotic equivalence:
$$(\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} \sim \frac{\mathbf{I}}{\lambda} 
\quad \quad \quad \quad \quad
\hat{\boldsymbol{\beta}}(\lambda) \sim \frac{(\mathbf{x}^\text{T} \mathbf{x}) \boldsymbol{\beta} + \mathbf{x}^\text{T} \boldsymbol{\varepsilon}}{\lambda}.$$
Note that using the Woodbury matrix-inverse formula gives the exact form:
$$\begin{align}
(\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} 
&= \frac{1}{\lambda} \bigg( \frac{\mathbf{x}^\text{T} \mathbf{x}}{\lambda} + \mathbf{I} \bigg)^{-1} \\[6pt]
&= \frac{1}{\lambda} \bigg( \mathbf{I} - \frac{1}{\lambda}  \mathbf{x}^\text{T} \bigg( \mathbf{I} + \frac{\mathbf{x} \mathbf{x}^\text{T}}{\lambda} \bigg) \mathbf{x} \bigg) \\[6pt]
&= \frac{1}{\lambda^3} \bigg( \lambda^2 \mathbf{I} - \lambda (\mathbf{x}^\text{T} \mathbf{x}) - (\mathbf{x}^\text{T} \mathbf{x})^2 \bigg), \\[6pt]
\end{align}$$
and this allows us to express the ridge-estimator (or its moments) in terms of matrix operations that do not involve inversion.

Variance and correlation: The variance matrix for the ridge estimator is given by:
$$\begin{align}
V(\lambda, \mathbf{x}) 
&\equiv \mathbb{V}(\hat{\boldsymbol{\beta}}(\lambda) | \mathbf{x}) \\[6pt]
&= (\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} \mathbf{x}^\text{T} \mathbb{V}(\boldsymbol{\varepsilon}) ((\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} \mathbf{x}^\text{T})^\text{T} \\[6pt]
&= \sigma^2 (\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} \mathbf{x}^\text{T} \mathbf{I} ((\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} \mathbf{x}^\text{T})^\text{T} \\[6pt]
&= \sigma^2 (\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} (\mathbf{x}^\text{T} \mathbf{x}) ((\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1})^\text{T} \\[6pt]
&= \sigma^2 (\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} (\mathbf{x}^\text{T} \mathbf{x}) (\mathbf{x}^\text{T} \mathbf{x} + \lambda \mathbf{I})^{-1} \\[6pt]
&= \frac{\sigma^2}{\lambda^6} \bigg( \lambda^2 \mathbf{I} - \lambda (\mathbf{x}^\text{T} \mathbf{x}) - (\mathbf{x}^\text{T} \mathbf{x})^2 \bigg) (\mathbf{x}^\text{T} \mathbf{x}) \bigg( \lambda^2 \mathbf{I} - \lambda (\mathbf{x}^\text{T} \mathbf{x}) - (\mathbf{x}^\text{T} \mathbf{x})^2 \bigg) \\[6pt]
&= \frac{\sigma^2}{\lambda^6} \begin{bmatrix}
\lambda^4 \mathbf{I} - \lambda^3 (\mathbf{x}^\text{T} \mathbf{x}) - \lambda^2 (\mathbf{x}^\text{T} \mathbf{x})^2 \\
- \lambda^3 (\mathbf{x}^\text{T} \mathbf{x}) + \lambda^2 (\mathbf{x}^\text{T} \mathbf{x})^2 + \lambda (\mathbf{x}^\text{T} \mathbf{x})^3 \\
- \lambda^2 (\mathbf{x}^\text{T} \mathbf{x})^2 + \lambda (\mathbf{x}^\text{T} \mathbf{x})^3 + (\mathbf{x}^\text{T} \mathbf{x})^4 \\
\end{bmatrix} (\mathbf{x}^\text{T} \mathbf{x}) \\[6pt]
&= \frac{\sigma^2}{\lambda^6} [ \lambda^4 (\mathbf{x}^\text{T} \mathbf{x}) - 2 \lambda^3 (\mathbf{x}^\text{T} \mathbf{x})^2 - \lambda^2 (\mathbf{x}^\text{T} \mathbf{x})^3  + 2 \lambda (\mathbf{x}^\text{T} \mathbf{x})^4 + (\mathbf{x}^\text{T} \mathbf{x})^5 ]. \\[6pt]
\end{align}$$
As $\lambda \rightarrow \infty$ you have the asymptotic equivalence:
$$V(\lambda, \mathbf{x}) \sim V^*(\lambda, \mathbf{x}) \equiv \frac{\sigma^2}{\lambda^2} (\mathbf{x}^\text{T} \mathbf{x}),$$
which has elements:
$$V_{i,j}^*(\lambda, \mathbf{x}) = \frac{\sigma^2}{\lambda^2} \sum_k x_{i,k} x_{j,k}.$$
Consequently, the asymptotic correlation between any two elements of the ridge-estimator is:
$$\begin{align}
\mathbb{Corr}(\hat{\beta}_i, \hat{\beta}_j |\mathbf{x}) 
\sim \frac{\sum_k x_{i,k} x_{j,k}}{\sqrt{(\sum_k x_{i,k} x_{i,k})(\sum_k x_{j,k} x_{j,k})}}. \\[6pt]
\end{align}$$
