# Under exactly what conditions is ridge regression able to provide an improvement over ordinary least squares regression?

Ridge regression estimates parameters $\boldsymbol \beta$ in a linear model $\mathbf y = \mathbf X \boldsymbol \beta$ by $$\hat{\boldsymbol \beta}_\lambda = (\mathbf X^\top \mathbf X + \lambda \mathbf I)^{-1} \mathbf X^\top \mathbf y,$$ where $\lambda$ is a regularization parameter. It is well-known that it often performs better than the OLS regression (with $\lambda=0$) when there are many correlated predictors.

An existence theorem for ridge regression says that there always exists a parameter $\lambda^* > 0$ such that mean-squared-error of $\hat{\boldsymbol \beta}_\lambda$ is strictly smaller than mean-squared-error of the OLS estimation $\hat{\boldsymbol \beta}_\mathrm{OLS}=\hat{\boldsymbol \beta}_0$. In other words, an optimal value of $\lambda$ is always non-zero. This was apparently first proven in Hoerl and Kennard, 1970 and is repeated in many lecture notes that I find online (e.g. here and here). My question is about the assumptions of this theorem:

1. Are there any assumptions about the covariance matrix $\mathbf X^\top \mathbf X$?

2. Are there any assumptions about dimensionality of $\mathbf X$?

In particular, is the theorem still true if predictors are orthogonal (i.e. $\mathbf X^\top \mathbf X$ is diagonal), or even if $\mathbf X^\top \mathbf X=\mathbf I$? And is it still true if there is only one or two predictors (say, one predictor and an intercept)?

If the theorem makes no such assumptions and remains true even in these cases, then why is ridge regression usually recommended only in the case of correlated predictors, and never (?) recommended for simple (i.e. not multiple) regression?

This is related to my question about Unified view on shrinkage: what is the relation (if any) between Stein's paradox, ridge regression, and random effects in mixed models?, but no answers there clarify this point until now.

• It appears all but the last question are directly addressed in the Hoerl & Kennard paper, especially in the first sentence of the Introduction and the first sentence of the Conclusions. The last question can be answered by noting the covariance between a constant vector and any single predictor is always zero, which allows one (in a standard way) to reduce $\mathbf{X^\prime X}$ to a $1\times 1$ matrix. – whuber Nov 6 '14 at 16:45
• Thanks, @whuber. I do believe that Hoerl & Kennard paper answers my questions (at least the technical ones) -- one should be able to follow the proof and check the assumptions (I have not done it yet). But I am not fully convinced by the sentences you are referring to. How is the first sentence of the Intro related to my question? The first sentence of the Conclusions does suggest that if $\mathbf X^\top \mathbf X$ has uniform spectrum (e.g. is equal to $\mathbf I$) then the theorem does not apply. But I am not 100% sure, as I don't see this assumption explicitly stated before the proof. – amoeba Nov 6 '14 at 21:25
• Look what kinds of questions can be asked by high rep users (who typically only answer them) (and likewise for your other linked question that sent me here stats.stackexchange.com/questions/122062/… ! – StephenBoesch Sep 17 '18 at 21:26

The answer to both 1 and 2 is no, but care is needed in interpreting the existence theorem.

## Variance of Ridge Estimator

Let $\hat{\beta^*}$ be the ridge estimate under penalty $k$, and let $\beta$ be the true parameter for the model $Y = X \beta + \epsilon$. Let $\lambda_1, \dotsc, \lambda_p$ be the eigenvalues of $X^T X$.
From Hoerl & Kennard equations 4.2-4.5, the risk, (in terms of the expected $L^2$ norm of the error) is

\begin{align*} E \left( \left[ \hat{\beta^*} - \beta \right]^T \left[ \hat{\beta^*} - \beta \right] \right)& = \sigma^2 \sum_{j=1}^p \lambda_j/ \left( \lambda_j +k \right)^2 + k^2 \beta^T \left( X^T X + k \mathbf{I}_p \right)^{-2} \beta \\ & = \gamma_1 (k) + \gamma_2(k) \\ & = R(k) \end{align*} where as far as I can tell, $\left( X^T X + k \mathbf{I}_p \right)^{-2} = \left( X^T X + k \mathbf{I}_p \right)^{-1} \left( X^T X + k \mathbf{I}_p \right)^{-1}.$ They remark that $\gamma_1$ has the interpretation of the variance of the inner product of $\hat{\beta^*} - \beta$, while $\gamma_2$ is the inner product of the bias.

Supposing $X^T X = \mathbf{I}_p$, then $$R(k) = \frac{p \sigma^2 + k^2 \beta^T \beta}{(1+k)^2}.$$ Let $$R^\prime (k) = 2\frac{k(1+k)\beta^T \beta - (p\sigma^2 + k^2 \beta^T \beta)}{(1+k)^3}$$ be the derivative of the risk w/r/t $k$. Since $\lim_{k \rightarrow 0^+} R^\prime (k) = -2p \sigma^2 < 0$, we conclude that there is some $k^*>0$ such that $R(k^*)<R(0)$.

The authors remark that orthogonality is the best that you can hope for in terms of the risk at $k=0$, and that as the condition number of $X^T X$ increases, $\lim_{k \rightarrow 0^+} R^\prime (k)$ approaches $- \infty$.

## Comment

There appears to be a paradox here, in that if $p=1$ and $X$ is constant, then we are just estimating the mean of a sequence of Normal$(\beta, \sigma^2)$ variables, and we know the the vanilla unbiased estimate is admissible in this case. This is resolved by noticing that the above reasoning merely provides that a minimizing value of $k$ exists for fixed $\beta^T \beta$. But for any $k$, we can make the risk explode by making $\beta^T \beta$ large, so this argument alone does not show admissibility for the ridge estimate.

## Why is ridge regression usually recommended only in the case of correlated predictors?

H&K's risk derivation shows that if we think that $\beta ^T \beta$ is small, and if the design $X^T X$ is nearly-singular, then we can achieve large reductions in the risk of the estimate. I think ridge regression isn't used ubiquitously because the OLS estimate is a safe default, and that the invariance and unbiasedness properties are attractive. When it fails, it fails honestly--your covariance matrix explodes. There is also perhaps a philosophical/inferential point, that if your design is nearly singular, and you have observational data, then the interpretation of $\beta$ as giving changes in $E Y$ for unit changes in $X$ is suspect--the large covariance matrix is a symptom of that.

But if your goal is solely prediction, the inferential concerns no longer hold, and you have a strong argument for using some sort of shrinkage estimator.

• Wow, thanks! Let me check my understanding of your "Comment" section: for any given $\beta$, an optimal $k$ is non-zero, but its value is different for different betas, and no fixed $k$ can beat $k=0$ for all betas, which is what is needed for admissibility. Correct? Apart from that, could you comment on my general question: [If the theorem makes no such assumptions, then] why is ridge regression usually recommended only for correlated predictors, and never recommended for simple (not multiple) regression? Is it because the positive effect is empirically known to be too small to bother? – amoeba Nov 7 '14 at 12:31
• H&K consistently assume $X^\prime X$ is of full rank. By stating that the answer to #1 is "no," are you claiming their results continue to be true when it is not? – whuber Nov 7 '14 at 16:26
• @whuber: Central to their derivation of the risk is that the ridge estimate $\hat{\beta^*} = Z \hat{\beta}$, where $\hat{\beta}$ is the OLS estimate and $Z = \left( (X^TX)^{-1} + k I_p \right)^{-1}$. This clearly cannot hold as such when $X^TX$ is rank deficient. But the OLS estimate doesn't exist--so perhaps any estimate with finite risk (take $k$ large enough and you'll get $\hat{\beta^*} \approx 0$, with risk $\beta^T \beta$) is better than an estimator that doesn't exist? As far as if the risk derivation still holds: I'm not sure. A different proof would be needed. – Andrew M Nov 8 '14 at 1:31
• @amoeba: yes, your restatement seems correct. To dominate the OLS estimator, we need some sort of adaptive procedure, in which $\lambda$ is a function of the data. On your other thread, Xi'an had a comment about adaptive ridge estimates, so that might be a place to look. RE: ridge estimates for orthogonal designs--I have added another comment as far as the guidance I'd take from their proof. – Andrew M Nov 8 '14 at 1:42