# Angle between $\hat{y}$ and $y$ stays the same as $\lambda$ in ridge regression is adjusted

I was given a thought experiment a while back to think about, but I haven't been able to come up with a solution.

The question is

For some dataset $$X$$ with response $$Y$$, you apply ridge regression. However, no matter what value of $$\lambda$$ you use, the angle between $$\hat{Y}$$ and $$Y$$ remains unchanged. What could be the source of this?

Based on how this question is posed, it seems that we can use a value for $$\lambda$$ and the angle between $$\hat{Y}$$ and $$Y$$ remain unchanged, so this means we can use $$\lambda = 0$$, which corresponds to our OLS solution. So what this seems to suggest is that no matter that $$\lambda$$, the $$\hat{Y}$$ determined from Ridge Regression is $$\hat{Y}$$ determined from OLS multiplied by some constant factor?

Is this the idea? If so, when does this occur?

I thought about a similar problem recently. I believe the necessary and sufficient condition (edit: on second thought, it may not be necessary, but it is a sufficient condition) is when the centered $$X$$ has orthonormal columns, in which case

$$\hat{\beta}_{ridge} = \frac{\hat{\beta}_{OLS}}{1+\lambda}$$

As you can see in this orthonormal case, $$\hat{\beta}_{ridge}$$ is simply scaled by some constant that's inversely proportional to $$\lambda$$. More importantly what this means is $$\hat{y}_{ridge} = X\hat{\beta}_{ridge} = X\frac{\hat{\beta}_{OLS}}{1 + \lambda} = \frac{\hat{y}_{OLS}}{1 + \lambda}$$

So you can see that your ridge solution is simply a scaled version of $$\hat{y}_{OLS}$$. Each direction of $$\hat{y}_{OLS}$$ is scaled by the same factor, so the angle between $$\hat{y}_{ridge}$$ and $$y$$ remains the same $$\forall \lambda$$.

Note for any 2 vectors $$x,y \in \mathbb{R}^n$$, we have $$\cos \theta_{x, y} = \frac{x^Ty}{||x||||y||}$$.

For our case, we have

$$\cos \theta_{y, \hat{y}_{ridge}} = \frac{y^T\hat{y}_{ridge}}{||y||||\hat{y}_{ridge}||} \\ = \frac{y^T\hat{y}_{OLS}}{||y||||\hat{y}_{OLS}||} \ \ \forall \lambda$$

since $$||\hat{y}_{ridge}|| = \left|\frac{1}{1 + \lambda} \right| ||\hat{y}_{OLS}||$$, a property of the p-norm.

• That makes sense! Any updates on the "on second thought" part? Are there other situations (non-orthonormal) in which the angle will not change? – roulette01 Sep 8 at 22:26
• @roulette01 It might be a necessary condition. I haven't had the chance to give it much thought – anonuser01 Sep 8 at 22:46