On this site is the following question which claims that the $L^2$ regularised OLS preserves the angle between $\hat Y$ and $Y$ irrespective of the value $\lambda$. I have not found any source that makes or proves this assertion, and furthermore the answer in this question eventually resorts to assumptions about standardisations in $X$.I know that $\hat\beta=(X^\mathsf{T}X+\lambda\mathbb{1})^{-1}X^\mathsf{T}Y$, so $\hat Y=X(X^\mathsf{T}X+\lambda\mathbb{1})^{-1}X^\mathsf{T}Y$ and $$\cos\theta=\frac{Y^\mathsf{T}\hat Y}{||Y||||\hat{Y}||}=\frac{YX(X^\mathsf{T}X+\lambda\mathbb{1})^{-1}X^\mathsf{T}Y}{||Y||||X(X^\mathsf{T}X+\lambda\mathbb{1})^{-1}X^\mathsf{T}Y||}$$ but this doesn't really help. Is the assertion true, and if so how do I get past my sticking point? Thanks!

  • 4
    $\begingroup$ The linked post isn't making a general statement about ridge regression, it's asking about what special circumstances can give rise to the described behavior. That's why the answer turns on $X$ being orthonormal. $\endgroup$
    – Sycorax
    Oct 12, 2021 at 15:00


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy