3
$\begingroup$

I have run a lasso regression on a dataset of 100 observations and 80 variables (using 10-fold cross-validation to find the minimum lambda subsequently used in the final model). The lasso regression found approximately 40 of the variables to have non-zero coefficients.

I wanted to check my model and therefore divided those 100 observations into two sets (70/30 - the idea being that I would have a train and test set) and ran a lasso regression on the 70. All coefficients calculated were 0 (except for the intercept) - a dramatically different result than in the first model using all 100 variables.

Confused, I ran another lasso regression on just the 30 observations and found 3 variables to have non-zero coefficients.

I assume that my drastically different results stem from the fact that the data I have does not do a good job of explaining the dependent variable, but perhaps there is a better explanation?

In case this is helpful - I am interested in using lasso regression for prediction.

Improve this question
$\endgroup$
2
  • $\begingroup$ Something sounds amiss. Did you randomly divide your data into the two groups of 70 and 30 observations? If so, I would expect the results to be somewhat similar to the results you got on the full 100 observations. Are you confident that there is not an error in the way you carried out the lasso or selected the lambda? $\endgroup$
    – Jean V. Adams
    Commented Aug 21, 2013 at 12:32
  • 2
    $\begingroup$ As soon as you have many fewer observations than coefficients your model is grossly undetermined: there is no reasonable expectation that the fits for the two groups would have much in common with each other or with the model on all observations. $\endgroup$
    – whuber ♦
    Commented Aug 17, 2017 at 19:23

2 Answers 2

Reset to default
3
$\begingroup$

Your data set is too small (too few data points) compared to your dimensions 100 v.s. 80. When you do cross validation, you are splitting your small dataset and get even smaller training and validation set. This learned model for the training set is most probably not be representative and may not fit the validation set well. Based on your random split, you may get different results.

Maybe try test subset of the variables and find certain variables set firstly. One brute force way is to test each of your 80 variables to filter out the insignificant factors (should not be hard) for this case.

Improve this answer
$\endgroup$
2
  • 5
    $\begingroup$ "Too small" is rather obvious, don't you think? Perhaps you could discuss why that has caused such discrepant results and--even more constructively--maybe you could propose a way to deal with the problem of cross-validating a model when there are many variables compared to observations. $\endgroup$
    – whuber ♦
    Commented Aug 17, 2017 at 19:24
  • $\begingroup$ For this specific problem I think "too small" explains everything. I don't think lasso will give you meaningful results when you have such a small set with >80 variables. When you do cross validation, you are splitting your small dataset and get even smaller training and validation set. This learned model for the training set is most probably not be representative and may not work well for the validation set. Based on your random split, you may get different results as well. Usually when that data set is large, and you have too many variables, you may consider doing dimension reduction first. $\endgroup$
    – Yixing Zhang
    Commented Aug 18, 2017 at 17:32
1
$\begingroup$

Are some features highly correlated ? According to this paper about elastic net:

  1. Lasso variable selection is poor when the number of feature is higher than the number of observations.
  2. Lasso variable selection is unstable when you have groups of features that are highly correlated.
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.