What are the assumptions of ridge regression and how to test them? - Cross Validated most recent 30 from stats.stackexchange.com 2019-06-26T08:03:34Z https://stats.stackexchange.com/feeds/question/169664 http://www.creativecommons.org/licenses/by-sa/3.0/rdf https://stats.stackexchange.com/q/169664 20 What are the assumptions of ridge regression and how to test them? akyves https://stats.stackexchange.com/users/85761 2015-09-01T14:07:10Z 2018-12-12T17:37:45Z <p>Consider the standard model for multiple regression $$Y=X\beta+\varepsilon$$ where $\varepsilon \sim \mathcal N(0, \sigma^2I_n)$, so normality, homoscedasticity and uncorrelatedness of errors all hold.</p> <p>Suppose that we perform a ridge regression, by adding the same small amount to all the elements of the diagonal of $X$:</p> <p>$$\beta_\mathrm{ridge}=[X'X+kI]^{-1}X'Y$$</p> <p>There are some values of $k$ for which the ridge coefficient has less mean squared error than those obtained by OLS, although $\beta_\mathrm{ridge}$ is a biased estimator of $\beta$. In practice, $k$ is obtained by cross-validation.</p> <p><strong>Here is my question: what are the assumptions underlying the ridge model?</strong> To be more concrete,</p> <ol> <li><p>Are all the assumptions of ordinary least square (OLS) valid with ridge regression?</p></li> <li><p>If yes to question 1, how do we test homoscedasticity and lack of autocorrelation with a biased estimator of $\beta$?</p></li> <li><p>Is there any work on testing other OLS assumptions (homoscedasticity and lack of autocorrelation) under ridge regression?</p></li> </ol> https://stats.stackexchange.com/questions/169664/-/183198#183198 18 Answer by amoeba for What are the assumptions of ridge regression and how to test them? amoeba https://stats.stackexchange.com/users/28666 2015-11-23T19:12:15Z 2015-11-23T22:35:50Z <h3>What is an <em>assumption</em> of a statistical procedure?</h3> <p>I am not a statistician and so this might be wrong, but I think the word "assumption" is often used quite informally and can refer to various things. To me, an "assumption" is, strictly speaking, something that only a theoretical result (theorem) can have.</p> <p>When people talk about assumptions of linear regression (<a href="https://stats.stackexchange.com/questions/16381">see here</a> for an in-depth discussion), they are usually referring to the <a href="https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem" rel="noreferrer">Gauss-Markov theorem</a> that says that <em>under assumptions</em> of uncorrelated, equal-variance, zero-mean errors, OLS estimate is BLUE, i.e. is unbiased and has minimum variance. Outside of the context of Gauss-Markov theorem, it is not clear to me what a "regression assumption" would even mean.</p> <p>Similarly, assumptions of a, say, one-sample t-test refer to the assumptions under which $t$-statistic is $t$-distributed and hence the inference is valid. It is not called a "theorem", but it is a clear mathematical result: if $n$ samples are normally distributed, then $t$-statistic will follow Student's $t$-distribution with $n-1$ degrees of freedom.</p> <h3>Assumptions of penalized regression techniques</h3> <p>Consider now any regularized regression technique: ridge regression, lasso, elastic net, principal components regression, partial least squares regression, etc. etc. The whole point of these methods is to make a <em>biased</em> estimate of regression parameters, and hoping to reduce the expected loss by exploiting the bias-variance trade-off.</p> <p>All of these methods include one or several regularization parameters and none of them has a definite rule for selecting the values of these parameter. The optimal value is usually found via some sort of cross-validation procedure, but there are various methods of cross-validation and they can yield somewhat different results. Moreover, it is not uncommon to invoke some additional rules of thumb in addition to cross-validation. As a result, the actual outcome $\hat \beta$ of any of these penalized regression methods is not actually fully defined by the method, but can depend on the analyst's choices.</p> <p>It is therefore not clear to me how there can be any theoretical optimality statement about $\hat \beta$, and so I am not sure that talking about "assumptions" (presence or absence thereof) of penalized methods such as ridge regression makes sense at all.</p> <h3>But what about the mathematical result that ridge regression always beats OLS?</h3> <p>Hoerl &amp; Kennard (1970) in <a href="https://stats.stackexchange.com/questions/122936">Ridge Regression: Biased Estimation for Nonorthogonal Problems</a> proved that there <em>always</em> exists a value of regularization parameter $\lambda$ such that ridge regression estimate of $\beta$ has a strictly smaller expected loss than the OLS estimate. It is a surprising result -- <a href="https://stats.stackexchange.com/questions/122936/">see here</a> for some discussion, but it only proves the existence of such $\lambda$, which will be dataset-dependent.</p> <p>This result does not actually require any assumptions and is always true, but it would be strange to claim that ridge regression does not have any assumptions.</p> <h3>Okay, but how do I know if I can apply ridge regression or not?</h3> <p>I would say that even if we cannot talk of assumptions, we can talk about <em>rules of thumb</em>. It is well-known that ridge regression tends to be most useful in case of multiple regression with correlated predictors. It is well-known that it tends to outperform OLS, often by a large margin. It will tend to outperform it even in the case of heteroscedasticity, correlated errors, or whatever else. So the simple rule of thumb says that if you have multicollinear data, ridge regression and cross-validation is a good idea. </p> <p>There are probably other useful rules of thumb and tricks of trade (such as e.g. what to do with gross outliers). But they are not assumptions.</p> <p>Note that for OLS regression one needs some assumptions for $p$-values to hold. In contrast, it is tricky to obtain $p$-values in ridge regression. If this is done at all, it is done by bootstrapping or some similar approach and again it would be hard to point at specific assumptions here because there are no mathematical guarantees.</p> https://stats.stackexchange.com/questions/169664/-/381693#381693 1 Answer by Emma for What are the assumptions of ridge regression and how to test them? Emma https://stats.stackexchange.com/users/144721 2018-12-12T17:37:45Z 2018-12-12T17:37:45Z <p>I would like to provide some input from the statistics perspective. If Y~N(Xb, sigma2*In), then the mean square error of b^ is </p> <pre><code>MSE(b^)=E(b^-b).T*(b^-b)=E(|b^-b|^2)=sigma2*trace(inv(X.T*X)) D(|b^-b|^2)=2*sigma4*trace((X.T*X)^(-2)) b^=inv(X.T*X)*X.T*Y </code></pre> <p>If X.T<em>X is approximately zero,then inv(X.T</em>X) will be very big. So the parameter estimate of b is not stable and can have the following problem.</p> <ol> <li>some absolute value of the parameter estimate is very big</li> <li>b has opposite positive or negative sign than expected.</li> <li>adding or removing variables or observations will make the parameter estimates changes dramatically.</li> </ol> <p>In order to make the ordinal least square estimate of b stable, we introduce the ridge regression by estimating the <code>b^(k)=inv(X.T*X+kI)*X.T*Y.</code> And we can prove that there is always a k that make the mean square error of </p> <pre><code>MSE(b^(k)) &lt; MSE(b^). </code></pre> <p>In machine learning, the ridge regression is called the L2 regularization and is to combat over-fitting problems caused by many features. </p>