2
$\begingroup$

First time asking question in StackExchange after being a long time lurker.

I am trying to analyze some simple data using R. I found the best lambda for Ridge regression using cross-validation, then I generate a ridge regression model with the best lambda with the training dataset, then calculate the mean square error (MSE) on the test dataset.

In my particular case, I found that ridge regression sometimes yields higher MSE than regular multiple linear regression (OLS) for some of the random seed that used to generate train-test split; however, the difference is not that much.

I am wondering is it possible for OLS to yield lower MSE than ridge regression? Because I think the cross-validation will generate a lambda that minimize MSE.

$\endgroup$
3
  • $\begingroup$ What is your sample size and the number of dimensions? $\endgroup$
    – amoeba
    Commented Feb 4, 2019 at 8:22
  • 7
    $\begingroup$ Because the test dataset is supposed to be independent of the training dataset (albeit similar in characteristics), there are no guarantees. $\endgroup$
    – whuber
    Commented Jun 7, 2022 at 20:55
  • $\begingroup$ This can help: stats.stackexchange.com/questions/487593/… $\endgroup$
    – markowitz
    Commented Jun 7, 2022 at 21:23

1 Answer 1

1
$\begingroup$

There exists a lambda such that the MSE of ridge is smaller than that of OLS.

The problem is how to find that lambda. Cross-validation is too blunt of an instrument for finding an optimal lambda. Actually, no method exists that guaranties an optimal lambda for ridge. If the design matrix is not collinear (see: https://www.mathworks.com/matlabcentral/fileexchange/60551-vif-x?s_tid=prof_contriblnk) then OLS is just as good. However, if the matrix is collinear then you should use ridge, even if your lambda is suboptimal.
Note that elastic net uses both L1 and L2 penalties. The L1, LASSO, drops the variables that are collinear unpredictably even with the same data. This would be in direct contradiction to Belsley, Kuh, and Welsch, Regression Diagnostics.

$\endgroup$
2
  • $\begingroup$ Allen's PRESS statistic (i.e. leave-one-out cross-validation) seems a pretty good way of choosing the ridge parameter, especially if you perform it in "canonical form", which makes it even cheaper to compute. $\endgroup$ Commented Jun 7, 2022 at 21:22
  • 4
    $\begingroup$ Re "should use ridge:" When the design matrix is collinear, you should first investigate the nature of the collinearity and make variable choices based on your subject-matter knowledge and objectives. This position is described in Belsley, Kuh, and Welsch, Regression Diagnostics. When this kind of investigation is impossible (perhaps because you have no understanding of the variables or there are far too many of them), then ridge regression becomes a useful option. But then why not go all the way and use Elastic Net? $\endgroup$
    – whuber
    Commented Jun 10, 2022 at 13:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.