3
$\begingroup$

I have run a lasso regression on a dataset of 100 observations and 80 variables (using 10-fold cross-validation to find the minimum lambda subsequently used in the final model). The lasso regression found approximately 40 of the variables to have non-zero coefficients.

I wanted to check my model and therefore divided those 100 observations into two sets (70/30 - the idea being that I would have a train and test set) and ran a lasso regression on the 70. All coefficients calculated were 0 (except for the intercept) - a dramatically different result than in the first model using all 100 variables.

Confused, I ran another lasso regression on just the 30 observations and found 3 variables to have non-zero coefficients.

I assume that my drastically different results stem from the fact that the data I have does not do a good job of explaining the dependent variable, but perhaps there is a better explanation?

In case this is helpful - I am interested in using lasso regression for prediction.

$\endgroup$
  • $\begingroup$ Something sounds amiss. Did you randomly divide your data into the two groups of 70 and 30 observations? If so, I would expect the results to be somewhat similar to the results you got on the full 100 observations. Are you confident that there is not an error in the way you carried out the lasso or selected the lambda? $\endgroup$ – Jean V. Adams Aug 21 '13 at 12:32
  • 2
    $\begingroup$ As soon as you have many fewer observations than coefficients your model is grossly undetermined: there is no reasonable expectation that the fits for the two groups would have much in common with each other or with the model on all observations. $\endgroup$ – whuber Aug 17 '17 at 19:23
1
$\begingroup$

Your data set is too small (too few data points) compared to your dimensions 100 v.s. 80. When you do cross validation, you are splitting your small dataset and get even smaller training and validation set. This learned model for the training set is most probably not be representative and may not fit the validation set well. Based on your random split, you may get different results.

Maybe try test subset of the variables and find certain variables set firstly. One brute force way is to test each of your 80 variables to filter out the insignificant factors (should not be hard) for this case.

$\endgroup$

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

  • 4
    $\begingroup$ "Too small" is rather obvious, don't you think? Perhaps you could discuss why that has caused such discrepant results and--even more constructively--maybe you could propose a way to deal with the problem of cross-validating a model when there are many variables compared to observations. $\endgroup$ – whuber Aug 17 '17 at 19:24
  • $\begingroup$ For this specific problem I think "too small" explains everything. I don't think lasso will give you meaningful results when you have such a small set with >80 variables. When you do cross validation, you are splitting your small dataset and get even smaller training and validation set. This learned model for the training set is most probably not be representative and may not work well for the validation set. Based on your random split, you may get different results as well. Usually when that data set is large, and you have too many variables, you may consider doing dimension reduction first. $\endgroup$ – Yixing Zhang Aug 18 '17 at 17:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.