Explaining Constant $\mathrm{corr}(\hat{y},y)$ in Ridge Regression as $\lambda$ Varies

Question

I was recently asked the question, "In ridge regression $\hat{y}=X(X^\top X+\lambda I)^{-1} X^\top y$, why might the correlation $\mathrm{corr}(\hat{y},y)$ between the predicted values ($\hat{y}$) and actual values ($y$) remain constant as $\lambda$ varies?" My initial thought is that this could be due to the eigenvalues of the $X^\top X$ matrix being roughly equal. However, I'm not certain if this is the only factor at play or if there are other potential reasons for this phenomenon. Thank you so much!

Answer 1

When you have a ridge in the parameter space, that ridge is a set of values where the fit hardly changes anywhere along it (the $\hat{y}$ values are essentially the same along that ridge even though $\hat{\beta}$ changes).

It's $\hat{\beta}$ that's not well-determined if you don't regularize, while the fit is nearly fixed.

Since that's the exact case where you want to use ridge regression, a collection of more or less regularized fits (larger or smaller $\lambda$ values) will still end up very near to the top of that ridge, and so have almost the same fit -- i.e. almost the same $\hat{y}$ values.

Which is to say, when you really want to use ridge regression (because you have an ill-determined system leading to a ridge in log-likelihood - or in -SSE - in the beta-space), that's exactly when you should expect to see the fit hardly change as you change $\lambda$.

Since the fit is barely changing, the correlation is nearly constant.