Angle between $\hat{y}$ and $y$ stays the same as $\lambda$ in ridge regression is adjusted

Question

I was given a thought experiment a while back to think about, but I haven't been able to come up with a solution.

The question is

For some dataset $X$ with response $Y$, you apply ridge regression. However, no matter what value of $\lambda$ you use, the angle between $\hat{Y}$ and $Y$ remains unchanged. What could be the source of this?

Based on how this question is posed, it seems that we can use a value for $\lambda$ and the angle between $\hat{Y}$ and $Y$ remain unchanged, so this means we can use $\lambda = 0$, which corresponds to our OLS solution. So what this seems to suggest is that no matter that $\lambda$, the $\hat{Y}$ determined from Ridge Regression is $\hat{Y}$ determined from OLS multiplied by some constant factor?

Is this the idea? If so, when does this occur?

Answer 1

I thought about a similar problem recently. I believe the necessary and sufficient condition (edit: on second thought, it may not be necessary, but it is a sufficient condition) is when the centered $X$ has orthonormal columns, in which case

$$ \hat{\beta}_{ridge} = \frac{\hat{\beta}_{OLS}}{1+\lambda} $$

As you can see in this orthonormal case, $\hat{\beta}_{ridge}$ is simply scaled by some constant that's inversely proportional to $\lambda$. More importantly what this means is $$ \hat{y}_{ridge} = X\hat{\beta}_{ridge} = X\frac{\hat{\beta}_{OLS}}{1 + \lambda} = \frac{\hat{y}_{OLS}}{1 + \lambda} $$

So you can see that your ridge solution is simply a scaled version of $\hat{y}_{OLS}$. Each direction of $\hat{y}_{OLS}$ is scaled by the same factor, so the angle between $\hat{y}_{ridge}$ and $y$ remains the same $\forall \lambda$.

Note for any 2 vectors $x,y \in \mathbb{R}^n$, we have $\cos \theta_{x, y} = \frac{x^Ty}{||x||||y||}$.

For our case, we have

$$ \cos \theta_{y, \hat{y}_{ridge}} = \frac{y^T\hat{y}_{ridge}}{||y||||\hat{y}_{ridge}||} \\ = \frac{y^T\hat{y}_{OLS}}{||y||||\hat{y}_{OLS}||} \ \ \forall \lambda $$

since $||\hat{y}_{ridge}|| = \left|\frac{1}{1 + \lambda} \right| ||\hat{y}_{OLS}||$, a property of the p-norm.