3
$\begingroup$

Before I start, I have already looked at the answers for related questions:

It seems from answers to these questions that when glmnet returns a lambda value where lambda_1se=lambda_min and where all its coefficients are zero, that simply indicates that X is just not predictive of Y?

However I find this very strange in my case because all the Xs (three variables, to be exact) that I am using have been shown in previous literature to differ significantly between the two groups (the Y) that I am comparing, and one of the Xs is a very well-established predictor of Y. Furthermore, when I run these variables as stepwise logistic regression on SPSS, I get very high AUCs for them.

Also, it is only the coefficients for the very first lambda from the run of lambdas that result in all-zero coefficients. From the second lambda on, the number of nonzero coefficients increase and it also seems that the AUCs at those lambda values remain pretty stable. So I am confused as to why glmnet would ignore those other lambda values that give similar performance, and choose the one that gives all-zeros, and which gives an AUC of 0.5 on my test set.

On a further note, the reason I am using glmnet on already well-established predictors of Y is to use the combination of these predictors as a "gold standard" or reference to which I will compare the results of my actual predictors of interest. In other words, I am doing glmnet on two separate sets of predictors and comparing their AUC.


Additional info:
A sample output cvglmnet is as below:

  • My sample size is 324, with 179 in negative class and 145 in positive class. The number of subjects in the image below is 291 because I am running nested cv and the number of outer loop training data is 291.
  • $\alpha$=0.5 here (it seems only alpha=0 results in AUC that is not 0.5).

  • 'object' in the left top image is the CVerr returned by cvglmnet.
  • 'object.glmnet_fit' in the right top image contains the parameters calculated at each lambda (for some reason, there are only 71 runs of lambda although I think the default number of runs should be 100).
  • 'object.glmnet_fit.beta' contains the beta values of the three predictor variables at each lambda value.

As you can see, the optimal lambda value chosen is the first lambda, which assigned all predictor variables zero coefficients.

$\endgroup$
  • $\begingroup$ lasso results are theoretically biased towards zero. You can try some of the nonconvex penalty parameters like MCP or SCAD which are less biased (R package ncvreg) $\endgroup$ – bdeonovic Apr 3 '17 at 12:51
  • $\begingroup$ Some questions: How large is your dataset? Are the classes imbalanced? Could you post the exact output of glmnet, together with the AUC values you calculated for each lambda? I assume you're using cv.glmnet directly? $\endgroup$ – AaronDefazio Apr 4 '17 at 4:03
  • $\begingroup$ Hi @AaronDefazio , I have added answers to your questions at the bottom of my question. As you can see, the AUC values for each lambda range from 0.8729~0.9323 ( as shown in object.cvm), however, when I apply the beta parameters at the optimal lambda value to my test dataset, I get all AUC=0.5...which is not surprising seeing that my chosen beta values are all zero. $\endgroup$ – Michelle Apr 4 '17 at 7:55
  • $\begingroup$ @Michelle I have a suggestion that might help. Could you try running with type.measure='auc' and with nfolds=3? $\endgroup$ – AaronDefazio Apr 5 '17 at 1:47
  • $\begingroup$ @AaronDefazio I am already running cvglmnet with type='auc'. I have tried making nfolds=3 like you said, but I still get the same results (AUC=0.5). Could you tell me what was the reason for your suggestion? $\endgroup$ – Michelle Apr 5 '17 at 6:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.