I am using the glmnet package in R to predict credit default. I have a 50 x variables which I have used in the first model.
fit1=cv.glmnet(x[1:test,1:50], y[1:test], type.measure="class")
I have also generated interaction action terms covering all possible two-way interactions between x variables (i.e. x1*x2, x1*x3... xn*xn). This adds approx 2000 variables to the data set. Some of these interactions I know to be significant above and beyond the impact of of their linear effects. I then used both the 50 original variable, plus the 2000 interactions to fit my second model.
fit2=cv.glmnet(x[1:test,], y[1:test], type.measure="class")
I figured fit2 would be atleast as good as fit1 in out of sample testing. But strangely, the the best out-of-sample classification rate from fit1 (tested across all values of lambda) beats the equivalent from fit2.
I would've though fit2 to perform at least as well and probably better than fit1 (given that the lasso algorithm should push out all non-important interactions, which is likely to be most of them). I could manually add in only the interaction terms I suspect to be important, but then I face the dilemma about where to draw the line and lose the feature selection capability, which is one of the most useful aspects of lasso regression. My questions:
- Is there a logical explanation as to why fit2 performs better out of sample than fit2?
- How can I improve fit1 by including interaction variables that I am confident are significant, without reducing predictive performance?