I am using the glmnet package in R to predict credit default. I have a 50 x variables which I have used in the first model.

fit1=cv.glmnet(x[1:test,1:50], y[1:test], type.measure="class")

I have also generated interaction action terms covering all possible two-way interactions between x variables (i.e. x1*x2, x1*x3... xn*xn). This adds approx 2000 variables to the data set. Some of these interactions I know to be significant above and beyond the impact of of their linear effects. I then used both the 50 original variable, plus the 2000 interactions to fit my second model.

 fit2=cv.glmnet(x[1:test,], y[1:test], type.measure="class")

I figured fit2 would be atleast as good as fit1 in out of sample testing. But strangely, the the best out-of-sample classification rate from fit1 (tested across all values of lambda) beats the equivalent from fit2.

I would've though fit2 to perform at least as well and probably better than fit1 (given that the lasso algorithm should push out all non-important interactions, which is likely to be most of them). I could manually add in only the interaction terms I suspect to be important, but then I face the dilemma about where to draw the line and lose the feature selection capability, which is one of the most useful aspects of lasso regression. My questions:

  1. Is there a logical explanation as to why fit2 performs better out of sample than fit2?
  2. How can I improve fit1 by including interaction variables that I am confident are significant, without reducing predictive performance?
  • $\begingroup$ Does the same behavior persist after cross validation as well? $\endgroup$ – Alex R. Apr 6 '16 at 0:23

First off, it looks like this is a classification problem, so make sure to have the type.measure option set to class, as such:

fit2=cv.glmnet(x[1:test,], y[1:test], type.measure = "class")

Remember that the Lasso loss function we try to minimize is the sum of the squared residuals plus lambda*(sum of the absolute value of the coefficient magnitudes, excluding the intercept). So, if you are comparing a lambda value for both models, they will keep approximately the same number of variables and similar magnitudes, because the cost for a large coefficient value is similar between the two models. However, when adding 2000 variables for which you want to include some in the model while also keeping your original significant variables, you need to adjust for a lower lambda, to be more inclusive.

If the some of the variables you are including are indeed significant, then the reason why your fit2 does not fit as well as fit1 is because the 2000 variables you are introducing may be valuable in predicting y, but not AS valuable as the variables in fit1. So, if the lambdas for both models are similar, the difference will be in sometimes including variables of the 2000 that are good but not as good as some of the originals for which they are replacing (but appear more important in the Lasso algorithm due to your training sample being slightly different than the population as a whole). With so many new variables added, the probability of randomly sampling where at least 1 of them appears more significant than it should be is high. In a shrinkage algorithm like the Lasso, this could seriously affect the results. Additionally, if some of the significant variables are highly correlated, then in a random sample some could go to zero if the correlated variable is more prevalent in the sample than in the population.

So, it is likely you want to change to the class measure if you haven't already, but besides that, it could be that the search for lambda is not including a small enough value for the fit2 model. Consider creating your own grid for lambda and and run the cv.glmnet with that grid. Here is an example version you can use:

grid = 10^seq(10, -2, length=100)
fit2=cv.glmnet(x[1:test,], y[1:test], type.measure = "class", lambda = grid)
| cite | improve this answer | |
  • $\begingroup$ Thank you for the advice about specifying the type as "class". I was already doing this, but didn't originally include in the post (now amended). Also the cv.glmnet algorithm automatically tests 100 values of lambda beginning with the small lambda that includes no variables and finishes with the largest lambda that includes all, so the issue is not that I am not testing small enough.The sweet spot seems to be around the 200-250 feature mark, but fit1 achieves a better out of sample prediction than fit2 at this point, which confounds me. $\endgroup$ – user3725021 Sep 23 '15 at 13:37
  • $\begingroup$ Just to clarify, it's the largest lambda that includes no variables and smaller lambdas would include more but not necessarily all. $\endgroup$ – gomfy Jun 16 at 20:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.