I ran three regularization methods, lasso, ridge, and elastic net. Lasso was able to get the best accuracy, so I'm selecting it. Is there a way to calculate odds ratio from the coefficients? Does it make sense to do it in glmnet?

I took the following steps:

train.control <- trainControl(method = "repeatedcv", 
                             number = 10, 
                             repeats = 5, 
                             allowParallel = T,
                             verboseIter = T)

lasso_model <- train(traget~ ., 
                trControl = train.control, 
                method = "glmnet",
                tuneGrid = expand.grid(alpha = 1, 
                                       lambda = seq(0.0001, 0.05, length = 5)),
                family = "binomial")

Plot and predict the model

plot(lasso_model$finalModel, xvar = "lambda", label = T)

plot(lasso_model$finalModel, xvar = "dev", label = T)

plot(varImp(lasso_model, scale = F))

p.lasso.pred <- predict(lasso_model, testTransformed)

p.lasso.pred.cm <- confusionMatrix(p.lasso.pred, testTransformed$BMK_R_Derailment, mode = "prec_recall")

Now, all tutorials that I've read stops at this point. I'm really confused as to whether to stop here, or take the features from lasso with coefficients > 0 and run logistic again to get the odds ratio for the coefficients.

And I also did that. However, most of the variables are not significant (which is fine). Then should I select the variables that are significant and do the regular (step-wise - not sure if I should do this) logistic regression? or leave the model as is because lasso produced those features?


If you ran glmnet with family="binomial" , the coefficients are log odds ratio, so exponential of these will give the odds ratio. You can check out their website where it writes (sorry I took a screen shot because of some symbols):

enter image description here

So for example:


fit = cv.glmnet(x=as.matrix(Pima.tr[,-ncol(Pima.tr)]),
y= Pima.tr$type,family="binomial",alpha=1)

Most likely you can follow the recommendation of 1se from best for the best lambda:

opt.lam = fit$lambda.1se
as.matrix(coef(fit, s = opt.lam))
(Intercept) -5.49752003
npreg        0.02408488
glu          0.02132413
bp           0.00000000
skin         0.00000000
bmi          0.03009826
ped          0.50900237
age          0.02462429

In this dataset, the coefficients indicate how much 1 unit of the predictors increase the log-odds of having type "yes" (or diabetes). You convert the above to log-odds by take the exponential.

| cite | improve this answer | |
  • $\begingroup$ @StupidWold, I'm really confused now. SO answer in stats.stackexchange.com/questions/428349/… stats that lasso coefficients should not be used for inference. So, does this mean we have to extract vars from lasso and run glm again? $\endgroup$ – user1828605 Apr 8 at 13:04
  • $\begingroup$ Ok, if you read that post, the OP wrote "I do not know how to get odds ratios with respective 95% CIs for the covariates retained" . So, if you need a 95% CI, you should not use the lasso coefficients. This is the inference part i guess $\endgroup$ – StupidWolf Apr 8 at 13:11
  • $\begingroup$ Can you elaborate on what you want to do? Because your question just says, "get odds ratio", and this is possible by just converting the coefficients to odds ratio $\endgroup$ – StupidWolf Apr 8 at 13:13
  • $\begingroup$ So, this means I can use the odds ratio as you showed above in my results? And, state that the odds of getting diabetes increases by a factor of exp(0.024) = 1.02 for 1 unit increase in npreg? $\endgroup$ – user1828605 Apr 8 at 13:17
  • $\begingroup$ Sorry if this is long winded. I don't want to tell you the wrong thing. Are you doing inference ? That is, are you trying to say something about the effects of your variable? For example in the above, if you are trying to say for example npreg has an effect, and it's increases the odds by ... , then most likely you should run the glm using the non-zero coefficients from lasso $\endgroup$ – StupidWolf Apr 8 at 13:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.