1
$\begingroup$

I am comparing OLS and LASSO regression for survey data. I have n>p, but I think my data is high-dimensional data as the p is 3000 and n is 48000. I am using k cross-validation. The results are quite surprising as from R-squared and MSE, the OLS performs much better than LASSO. I have read several discussions and papers previously, but still, I can't figure out why...

$\endgroup$
4
  • 2
    $\begingroup$ are you looking at in-sample or out-of-sample performance? $\endgroup$
    – Ben Bolker
    Apr 29, 2022 at 14:32
  • $\begingroup$ both. results for both samples are showing that OLS perform better $\endgroup$
    – Retno M
    Apr 29, 2022 at 14:48
  • $\begingroup$ Why is it surprising? Have you considered how your OLS is estimated in the n<<p regime? Also, the No Free-lunch Theorem applies. $\endgroup$
    – Firebug
    Apr 29, 2022 at 14:55
  • $\begingroup$ sorry, I corrected my question. p=3000 and n=48000. so it is n>p. But I think I still can categorize it as high-dimensional data, correct me if I'm wrong. One that I'm aware is LASSO will perform better than linear regression in high-dimensional data through regularization/shrinkage. But I don't find it in my case. $\endgroup$
    – Retno M
    Apr 29, 2022 at 15:08

1 Answer 1

2
$\begingroup$

This is one example of a "curse of dimensionality" described for LASSO in An Introduction to Statistical Learning, second edition, at the end of Chapter 6.

With a large number of potential predictors and LASSO forced to choose a comparatively small number among them, there's an increasing risk as p increases that some truly insignificant predictors might by chance have an association with outcome in a particular data sample and be included erroneously. That might also lead to omission of truly significant predictors and poor generalizability.

With OLS in this scenario, all of your potentially important predictors are available to the final model. With a 16:1 n:p ratio, that probably isn't at much risk of overfitting. For prediction, throwing away predictors (even "insignificant" ones) doesn't typically do much good unless you are overfitting. So OLS wins here.

$\endgroup$
2
  • $\begingroup$ Would you expect Ridge regression to perform better since it would not completely remove predictors from the model? $\endgroup$
    – Eli
    Apr 29, 2022 at 17:02
  • $\begingroup$ @Eli better than LASSO but maybe not better than OLS in this particular case, as you might not need any penalization at all with your fairly high n:p ratio. Penalization (ridge or LASSO) trades off an increase of bias for a decrease in variance when applied to new cases. If your model isn't overfitting and thus has low variance on new cases, there is no need for that tradeoff. In general, however, ridge can do better than LASSO for predictions if you don't need to cut down on the number of predictors and many are outcome-related, or than OLS when you are close to overfitting. Try it and see. $\endgroup$
    – EdM
    Apr 29, 2022 at 17:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.