# Interpreting glmnet penalty factor wrt unsatisfactory LASSO feature-selection?

In my dataset, I already know there is a feature; that is predictive of the outcome. However, I would like to know what other features are predictive.

Therefore, I constructed two glmnet models and would like to understand how I can interpret the effect of penalty factor on the result.

In the first model, all variables are treated equally meaning the corresponding penalty factor for all variables is set to 1. Here the model selects q number of features which are consistently selected across all ten-fold cross validation (including the variable that I already know is important).

In the second model, I set the penalty factor corresponding to the predictive feature (the one I already have a prior knowledge). Surprisingly, LASSO does not select any other feature than the one I expect consistently across all cross-validation folds.

From one side, I am more happy about the first model, because there are a few features that are selected along with the predictive feature and I can start believing they are predictive. However, from the other side, I am not happy because I have not used my prior knowledge.

• Penalty factor of 1 does not the penalize coefficient therefore the variable will always be in the model. It is because, I already have a prior knowledge on one the variable and I know it is predictive as I explained. I don't see how the idea of elastic net can be related to this question. – user4581 Jan 3 '17 at 18:02
• I am using glment in R; please have a look web.stanford.edu/~hastie/glmnet/glmnet_alpha.html and search for "penalty.factor". - penalty.factor is neither alpha nor lambda in lasso/glment setup. it is basically a way to incorporate prior information in the model. I am sure you can find more information about it in the package documentation :-) – user4581 Jan 3 '17 at 20:20
• I'm sorry (again), I was mixing up my terminology from the ISLR textbook. I'll clean up the above mistakes. – Upper_Case Jan 3 '17 at 20:48
• no problem - we are all here to learn :-) – user4581 Jan 4 '17 at 22:11
• penalty.factor is trying to kludge the model, which I've never seen needed. Please first try all the steps in my answer. Then post us your code. Also the plot.cv.glmnet() of deviance/log(lambda) plot around the knee. Use 5 steps per lambda decade. Really I think the title should be "Why is LASSO always only choosing one variable even though I suspect the other variables may be predictive?" – smci Feb 2 '17 at 4:23

LASSO means you're arbitrarily choosing glmnet(..., alpha=1) and is pretty harsh at eliminating variables (also it can suffer from stability issues esp. for small datasets with high variance between CV folds). LASSO/ alpha=1 is just asking for trouble in feature-selection, IME.

1. Try running with various alpha<1 values ("elastic-net") and you should see more variables used. In particular, alpha=0 is ridge-regression which is much less harsh.

2. Make sure you pass a full lambda sequence, not a single or default lambda value.

3. In the unlikely case you still don't see any other variables selected, then Run glmnet excluding your most predictive feature from the x-matrix. That was my first reaction. Tell us what you get. That's pretty much forcing elnet to pick the next-most-predictive variable(s). (Or you could try setting penalty.factor=0 on your most-predictive variable).

4. Make sure you normalized the variables beforehand, using scale(). Otherwise it could simply be that your main feature happens to have a much smaller magnitude than the others, resulting in LASSO eliminating them, especially at some sharp knee in the deviance/log(lambda) plot. (Also, be sure to use enough lambda value steps to capture behavior around that knee; like 5 per decade. Post us the plot.cv.glmnet() around that knee, if you can. Maybe your lambda sequence is simply too coarse, e.g. <=2 steps/decade)

5. Repeat with cv.glmnet(). Use an explicit random-seed (set.seed()) to ensure reproducibility. Also try multiple random-seeds (=> fold selection) and see how stable(/unstable) it is wrt those. That should fix it, tell us what you experience. (EDIT: make sure to use the same seed(/set of seeds) for each alpha value, obviously; to ensure the folds will be identical).

6. You didn't say anything about the response variable, whether it's regression/classification (type.measure), the family (binomial?/gaussian?/etc.) or the CV loss-function (MSE/deviance/MAE/class?) It would help if you posted a snippet of code and data.

• 1. Not only is ridge regression less harsh, but it will not delete any features at all. – gammer Feb 2 '17 at 4:28
• @gammer: Yes, but Ridge can still downweight them. General Elastic-net certainly can still zero out features. OP should try multiple alpha values < 1 until they get more smooth variable-selection behavior. (Anyone got a rule-of-thumb on sweeping alpha value? I've never seen one. Other than "alpha=1 is a terrible default") – smci Feb 2 '17 at 4:43
• I don't know a rule of thumb but one option could be to move across a fine grid of values of alpha and jointly optimize the CV loss function as a function of both alpha and lambda. Of course, care must be taken to make sure the same folds are used for each value of alpha (or, better yet, do leave-one-out CV so that's a non-issue...of course, that's only an feasible option if the sample size isn't too big) – gammer Feb 2 '17 at 4:48
• for (alpha in c(0,.05,.1,.2,.3,.5,.7,.8,.9,.95,1.) { ... } should do ok. – smci Feb 2 '17 at 4:51
• Maybe it is overkill, but I think I got that from a web vignette written by Jerome Friedman and Trevor Hastie. Your approach makes sense. And yes, I agree that I've never seen a reason to mess with penalty.factor. I upvoted your answer. – gammer Feb 2 '17 at 5:08