Interpreting glmnet penalty factor wrt unsatisfactory LASSO feature-selection?

Question

In my dataset, I already know there is a feature; that is predictive of the outcome. However, I would like to know what other features are predictive.

Therefore, I constructed two glmnet models and would like to understand how I can interpret the effect of penalty factor on the result.

In the first model, all variables are treated equally meaning the corresponding penalty factor for all variables is set to 1. Here the model selects q number of features which are consistently selected across all ten-fold cross validation (including the variable that I already know is important).

In the second model, I set the penalty factor corresponding to the predictive feature (the one I already have a prior knowledge). Surprisingly, LASSO does not select any other feature than the one I expect consistently across all cross-validation folds.

From one side, I am more happy about the first model, because there are a few features that are selected along with the predictive feature and I can start believing they are predictive. However, from the other side, I am not happy because I have not used my prior knowledge.

Answer 1

LASSO means you're arbitrarily choosing glmnet(..., alpha=1) and is pretty harsh at eliminating variables (also it can suffer from stability issues esp. for small datasets with high variance between CV folds). LASSO/ alpha=1 is just asking for trouble in feature-selection, IME.

1. Try running with various alpha<1 values ("elastic-net") and you should see more variables used. In particular, alpha=0 is ridge-regression which is much less harsh.

2. Make sure you pass a full lambda sequence, not a single or default lambda value.

3. In the unlikely case you still don't see any other variables selected, then Run glmnet excluding your most predictive feature from the x-matrix. That was my first reaction. Tell us what you get. That's pretty much forcing elnet to pick the next-most-predictive variable(s). (Or you could try setting penalty.factor=0 on your most-predictive variable).

4. Make sure you normalized the variables beforehand, using scale(). Otherwise it could simply be that your main feature happens to have a much smaller magnitude than the others, resulting in LASSO eliminating them, especially at some sharp knee in the deviance/log(lambda) plot. (Also, be sure to use enough lambda value steps to capture behavior around that knee; like 5 per decade. Post us the plot.cv.glmnet() around that knee, if you can. Maybe your lambda sequence is simply too coarse, e.g. <=2 steps/decade)

5. Repeat with cv.glmnet(). Use an explicit random-seed (set.seed()) to ensure reproducibility. Also try multiple random-seeds (=> fold selection) and see how stable(/unstable) it is wrt those. That should fix it, tell us what you experience. (EDIT: make sure to use the same seed(/set of seeds) for each alpha value, obviously; to ensure the folds will be identical).

6. You didn't say anything about the response variable, whether it's regression/classification (type.measure), the family (binomial?/gaussian?/etc.) or the CV loss-function (MSE/deviance/MAE/class?) It would help if you posted a snippet of code and data.