In my dataset, I already know there is a feature; that is predictive of the outcome. However, I would like to know what other features are predictive.
Therefore, I constructed two glmnet models and would like to understand how I can interpret the effect of penalty factor on the result.
In the first model, all variables are treated equally meaning the corresponding penalty factor for all variables is set to 1. Here the model selects q number of features which are consistently selected across all ten-fold cross validation (including the variable that I already know is important).
In the second model, I set the penalty factor corresponding to the predictive feature (the one I already have a prior knowledge). Surprisingly, LASSO does not select any other feature than the one I expect consistently across all cross-validation folds.
From one side, I am more happy about the first model, because there are a few features that are selected along with the predictive feature and I can start believing they are predictive. However, from the other side, I am not happy because I have not used my prior knowledge.
penalty.factor
is trying to kludge the model, which I've never seen needed. Please first try all the steps in my answer. Then post us your code. Also theplot.cv.glmnet()
of deviance/log(lambda) plot around the knee. Use 5 steps per lambda decade. Really I think the title should be "Why is LASSO always only choosing one variable even though I suspect the other variables may be predictive?" $\endgroup$