0
$\begingroup$

I was given a thought experiment a while back to think about, but I haven't been able to come up with a solution.

The question is

For some dataset $X$ with response $Y$, you apply ridge regression. However, no matter what value of $\lambda$ you use, the angle between $\hat{Y}$ and $Y$ remains unchanged. What could be the source of this?

Based on how this question is posed, it seems that we can use a value for $\lambda$ and the angle between $\hat{Y}$ and $Y$ remain unchanged, so this means we can use $\lambda = 0$, which corresponds to our OLS solution. So what this seems to suggest is that no matter that $\lambda$, the $\hat{Y}$ determined from Ridge Regression is $\hat{Y}$ determined from OLS multiplied by some constant factor?

Is this the idea? If so, when does this occur?

$\endgroup$
2
$\begingroup$

I thought about a similar problem recently. I believe the necessary and sufficient condition (edit: on second thought, it may not be necessary, but it is a sufficient condition) is when the centered $X$ has orthonormal columns, in which case

$$ \hat{\beta}_{ridge} = \frac{\hat{\beta}_{OLS}}{1+\lambda} $$

As you can see in this orthonormal case, $\hat{\beta}_{ridge}$ is simply scaled by some constant that's inversely proportional to $\lambda$. More importantly what this means is $$ \hat{y}_{ridge} = X\hat{\beta}_{ridge} = X\frac{\hat{\beta}_{OLS}}{1 + \lambda} = \frac{\hat{y}_{OLS}}{1 + \lambda} $$

So you can see that your ridge solution is simply a scaled version of $\hat{y}_{OLS}$. Each direction of $\hat{y}_{OLS}$ is scaled by the same factor, so the angle between $\hat{y}_{ridge}$ and $y$ remains the same $\forall \lambda$.

Note for any 2 vectors $x,y \in \mathbb{R}^n$, we have $\cos \theta_{x, y} = \frac{x^Ty}{||x||||y||}$.

For our case, we have

$$ \cos \theta_{y, \hat{y}_{ridge}} = \frac{y^T\hat{y}_{ridge}}{||y||||\hat{y}_{ridge}||} \\ = \frac{y^T\hat{y}_{OLS}}{||y||||\hat{y}_{OLS}||} \ \ \forall \lambda $$

since $||\hat{y}_{ridge}|| = \left|\frac{1}{1 + \lambda} \right| ||\hat{y}_{OLS}||$, a property of the p-norm.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ That makes sense! Any updates on the "on second thought" part? Are there other situations (non-orthonormal) in which the angle will not change? $\endgroup$ – roulette01 Sep 8 at 22:26
  • $\begingroup$ @roulette01 It might be a necessary condition. I haven't had the chance to give it much thought $\endgroup$ – anonuser01 Sep 8 at 22:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.