21
$\begingroup$

Consider the standard model for multiple regression $$Y=X\beta+\varepsilon$$ where $\varepsilon \sim \mathcal N(0, \sigma^2I_n)$, so normality, homoscedasticity and uncorrelatedness of errors all hold.

Suppose that we perform a ridge regression, by adding the same small amount to all the elements of the diagonal of $X$:

$$\beta_\mathrm{ridge}=[X'X+kI]^{-1}X'Y$$

There are some values of $k$ for which the ridge coefficient has less mean squared error than those obtained by OLS, although $\beta_\mathrm{ridge}$ is a biased estimator of $\beta$. In practice, $k$ is obtained by cross-validation.

Here is my question: what are the assumptions underlying the ridge model? To be more concrete,

  1. Are all the assumptions of ordinary least square (OLS) valid with ridge regression?

  2. If yes to question 1, how do we test homoscedasticity and lack of autocorrelation with a biased estimator of $\beta$?

  3. Is there any work on testing other OLS assumptions (homoscedasticity and lack of autocorrelation) under ridge regression?

$\endgroup$
  • 6
    $\begingroup$ Please note that OLS does not assume predictors are independent. It is only certain particular solution methods or formulas that make such assumptions. What is of importance is how you select the ridge regression multiplier, not that the estimate of $\beta$ might be biased. If that multiplier is selected by eyeballing a ridge trace, then you don't really have a way to quantify uncertainties, which calls into question most of the formal diagnostic tests in linear regression theory. This leads me ask what you actually mean by "ridge regression": how exactly are you estimating its parameter? $\endgroup$ – whuber Sep 1 '15 at 14:13
  • $\begingroup$ Perhaps I am wrong, but considering the standard model of multiple regression $\beta_{OLS}=(X'X)^{-1}X'Y$. And if $X$ is not full rank, this lead to a non invertible matrix $X'X$, especially in case of high dimension of X. I have edited my question. Thanks. $\endgroup$ – akyves Sep 2 '15 at 6:55
  • 1
    $\begingroup$ Linear regression can perfectly deal with collinearity, as long as it is not "too large". $\endgroup$ – jona Sep 2 '15 at 10:47
  • 3
    $\begingroup$ That's not the model for multiple regression: it's only one way of expressing the least squares estimate. When $X^\prime X$ is not invertible, the normal equations still have solutions and (usually) the model still has a unique fit, which means it makes unique predictions. $\endgroup$ – whuber Sep 2 '15 at 14:06
  • $\begingroup$ Related: Model assumptions of partial least squares (PLS) regression. $\endgroup$ – amoeba says Reinstate Monica Nov 20 '15 at 1:13
21
+100
$\begingroup$

What is an assumption of a statistical procedure?

I am not a statistician and so this might be wrong, but I think the word "assumption" is often used quite informally and can refer to various things. To me, an "assumption" is, strictly speaking, something that only a theoretical result (theorem) can have.

When people talk about assumptions of linear regression (see here for an in-depth discussion), they are usually referring to the Gauss-Markov theorem that says that under assumptions of uncorrelated, equal-variance, zero-mean errors, OLS estimate is BLUE, i.e. is unbiased and has minimum variance. Outside of the context of Gauss-Markov theorem, it is not clear to me what a "regression assumption" would even mean.

Similarly, assumptions of a, say, one-sample t-test refer to the assumptions under which $t$-statistic is $t$-distributed and hence the inference is valid. It is not called a "theorem", but it is a clear mathematical result: if $n$ samples are normally distributed, then $t$-statistic will follow Student's $t$-distribution with $n-1$ degrees of freedom.

Assumptions of penalized regression techniques

Consider now any regularized regression technique: ridge regression, lasso, elastic net, principal components regression, partial least squares regression, etc. etc. The whole point of these methods is to make a biased estimate of regression parameters, and hoping to reduce the expected loss by exploiting the bias-variance trade-off.

All of these methods include one or several regularization parameters and none of them has a definite rule for selecting the values of these parameter. The optimal value is usually found via some sort of cross-validation procedure, but there are various methods of cross-validation and they can yield somewhat different results. Moreover, it is not uncommon to invoke some additional rules of thumb in addition to cross-validation. As a result, the actual outcome $\hat \beta$ of any of these penalized regression methods is not actually fully defined by the method, but can depend on the analyst's choices.

It is therefore not clear to me how there can be any theoretical optimality statement about $\hat \beta$, and so I am not sure that talking about "assumptions" (presence or absence thereof) of penalized methods such as ridge regression makes sense at all.

But what about the mathematical result that ridge regression always beats OLS?

Hoerl & Kennard (1970) in Ridge Regression: Biased Estimation for Nonorthogonal Problems proved that there always exists a value of regularization parameter $\lambda$ such that ridge regression estimate of $\beta$ has a strictly smaller expected loss than the OLS estimate. It is a surprising result -- see here for some discussion, but it only proves the existence of such $\lambda$, which will be dataset-dependent.

This result does not actually require any assumptions and is always true, but it would be strange to claim that ridge regression does not have any assumptions.

Okay, but how do I know if I can apply ridge regression or not?

I would say that even if we cannot talk of assumptions, we can talk about rules of thumb. It is well-known that ridge regression tends to be most useful in case of multiple regression with correlated predictors. It is well-known that it tends to outperform OLS, often by a large margin. It will tend to outperform it even in the case of heteroscedasticity, correlated errors, or whatever else. So the simple rule of thumb says that if you have multicollinear data, ridge regression and cross-validation is a good idea.

There are probably other useful rules of thumb and tricks of trade (such as e.g. what to do with gross outliers). But they are not assumptions.

Note that for OLS regression one needs some assumptions for $p$-values to hold. In contrast, it is tricky to obtain $p$-values in ridge regression. If this is done at all, it is done by bootstrapping or some similar approach and again it would be hard to point at specific assumptions here because there are no mathematical guarantees.

$\endgroup$
  • $\begingroup$ In the situation where one is deriving properties of inference in relation to some procedure, whether it is properties of a hypothesis test of a regression slope or properties of a confidence interval or a prediction interval, for example, the tests themselves will be derived under some set of assumptions. Since in many subject areas by far the most common purpose of using regression is to perform some kind of inference (indeed, in some application areas it's rarely done for any other reason), the assumptions that would be made for the inferential procedure are naturally associated with ...ctd $\endgroup$ – Glen_b -Reinstate Monica Nov 26 '15 at 6:42
  • $\begingroup$ ctd... the thing they're used on. So if you need some assumptions to derive a t-test for testing a regression coefficient or for a partial F test or for a CI for the mean or a prediction interval ... and the usual forms of inference all make the same or almost the same collection of assumptions, then those would reasonably be considered as assumptions associated with performing inference using that thing. If one is to perform any inference with ridge regression (say a prediction interval) and makes assumptions in order to do so, those might equally be said to be assumptions ... ctd $\endgroup$ – Glen_b -Reinstate Monica Nov 26 '15 at 6:45
  • $\begingroup$ needed to be able to derive (and presumably, then, to use) that particular kind of inference on ridge regression. $\endgroup$ – Glen_b -Reinstate Monica Nov 26 '15 at 6:46
  • $\begingroup$ I don't think I disagree with anything you said, @Glen_b. I guess my point is that I don't know what sort of inference people perform with ridge regression... Even to get a p-value for ridge regression coefficient is a nontrivial, and non-standard, task. To me, the "standard" usage of ridge regression is to apply cross-validation to measure its performance, e.g. cross-validated $R^2$. Is it considered inference? I guess so, but then it's very difficult even to give an interval around this point estimate; estimating variance of cross-validation estimators is notoriously tricky. [cont]. $\endgroup$ – amoeba says Reinstate Monica Nov 26 '15 at 10:46
  • 1
    $\begingroup$ Not too late I hope to say thanks @amoeba. Great answer! $\endgroup$ – akyves Mar 13 '17 at 22:49
1
$\begingroup$

I would like to provide some input from the statistics perspective. If Y~N(Xb, sigma2*In), then the mean square error of b^ is

MSE(b^)=E(b^-b).T*(b^-b)=E(|b^-b|^2)=sigma2*trace(inv(X.T*X))

D(|b^-b|^2)=2*sigma4*trace((X.T*X)^(-2))

b^=inv(X.T*X)*X.T*Y

If X.TX is approximately zero,then inv(X.TX) will be very big. So the parameter estimate of b is not stable and can have the following problem.

  1. some absolute value of the parameter estimate is very big
  2. b has opposite positive or negative sign than expected.
  3. adding or removing variables or observations will make the parameter estimates changes dramatically.

In order to make the ordinal least square estimate of b stable, we introduce the ridge regression by estimating the b^(k)=inv(X.T*X+kI)*X.T*Y. And we can prove that there is always a k that make the mean square error of

MSE(b^(k)) < MSE(b^).

In machine learning, the ridge regression is called the L2 regularization and is to combat over-fitting problems caused by many features.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.