Before I start, I have already looked at the answers for related questions:
- How to interpret all zero coefficients in the results of cv.glmnet?
- Why is cv.glmnet giving a lambda.min that is clearly not the lambda for minimum error?
It seems from answers to these questions that when glmnet returns a lambda value where lambda_1se=lambda_min and where all its coefficients are zero, that simply indicates that X is just not predictive of Y?
However I find this very strange in my case because all the Xs (three variables, to be exact) that I am using have been shown in previous literature to differ significantly between the two groups (the Y) that I am comparing, and one of the Xs is a very well-established predictor of Y. Furthermore, when I run these variables as stepwise logistic regression on SPSS, I get very high AUCs for them.
Also, it is only the coefficients for the very first lambda from the run of lambdas that result in all-zero coefficients. From the second lambda on, the number of nonzero coefficients increase and it also seems that the AUCs at those lambda values remain pretty stable. So I am confused as to why glmnet would ignore those other lambda values that give similar performance, and choose the one that gives all-zeros, and which gives an AUC of 0.5 on my test set.
On a further note, the reason I am using glmnet on already well-established predictors of Y is to use the combination of these predictors as a "gold standard" or reference to which I will compare the results of my actual predictors of interest. In other words, I am doing glmnet on two separate sets of predictors and comparing their AUC.
A sample output cvglmnet is as below:
- My sample size is 324, with 179 in negative class and 145 in positive class. The number of subjects in the image below is 291 because I am running nested cv and the number of outer loop training data is 291.
- $\alpha$=0.5 here (it seems only alpha=0 results in AUC that is not 0.5).
- 'object' in the left top image is the CVerr returned by cvglmnet.
- 'object.glmnet_fit' in the right top image contains the parameters calculated at each lambda (for some reason, there are only 71 runs of lambda although I think the default number of runs should be 100).
- 'object.glmnet_fit.beta' contains the beta values of the three predictor variables at each lambda value.
As you can see, the optimal lambda value chosen is the first lambda, which assigned all predictor variables zero coefficients.