I have run a lasso regression on a dataset of 100 observations and 80 variables (using 10-fold cross-validation to find the minimum lambda subsequently used in the final model). The lasso regression found approximately 40 of the variables to have non-zero coefficients.
I wanted to check my model and therefore divided those 100 observations into two sets (70/30 - the idea being that I would have a train and test set) and ran a lasso regression on the 70. All coefficients calculated were 0 (except for the intercept) - a dramatically different result than in the first model using all 100 variables.
Confused, I ran another lasso regression on just the 30 observations and found 3 variables to have non-zero coefficients.
I assume that my drastically different results stem from the fact that the data I have does not do a good job of explaining the dependent variable, but perhaps there is a better explanation?
In case this is helpful - I am interested in using lasso regression for prediction.