We have a dataset with around 20,000 variables and only 200 observations.

Our Naive Modelling:

  • We split it into train set (=150 observations) and validation set (=50 observations) and fit Linear regression.
  • Results / Train set: As expected, we get $R^2$ as 1 and $MSE$ as ~0 for train set.
  • Results / Validation set: As expected, validation set's results are horrible ($R^2$ = -854).

Is there a better way to model (such under-determined system as ours)?

  • 1
    $\begingroup$ Regularized regressions sometimes allow us to deal with this problem more productively than OLS, but as is so often the case, they need to be used rather carefully based on your context. You might find some of the answers here useful: stats.stackexchange.com/questions/407563/… $\endgroup$ Aug 3 at 14:35
  • 1
    $\begingroup$ Louis is totally right. But if you just want to get started quick run ridge regression aka tikonov regularization. It may not be optimal but it will give you a positive R2 ;) $\endgroup$ Aug 3 at 14:52


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.