There seems to be a lot of confusion in the comparison of using
caret to search for an optimal lambda and using
cv.glmnet to do the same task.
Many questions were posed, e.g.:
but no answer has been given, which might be due to the reproducability of the question. Following the first question, I give a quite similar example but do have the same question: Why are the estimated lambdas so different?
library(caret) library(glmnet) set.seed(849) training <- twoClassSim(50, linearVars = 2) set.seed(849) testing <- twoClassSim(500, linearVars = 2) trainX <- training[, -ncol(training)] testX <- testing[, -ncol(testing)] trainY <- training$Class # Using glmnet to directly perform CV set.seed(849) cvob1=cv.glmnet(x=as.matrix(trainX),y=trainY,family="binomial",alpha=1, type.measure="auc", nfolds = 3,lambda = seq(0.001,0.1,by = 0.001),standardize=FALSE) cbind(cvob1$lambda,cvob1$cvm) # best parameter cvob1$lambda.mi # best coefficient coef(cvob1, s = "lambda.min") # Using caret to perform CV cctrl1 <- trainControl(method="cv", number=3, returnResamp="all",classProbs=TRUE,summaryFunction=twoClassSummary) set.seed(849) test_class_cv_model <- train(trainX, trainY, method = "glmnet", trControl = cctrl1,metric = "ROC", tuneGrid = expand.grid(alpha = 1,lambda = seq(0.001,0.1,by = 0.001))) test_class_cv_model # best parameter test_class_cv_model$bestTune # best coefficient coef(test_class_cv_model$finalModel, test_class_cv_model$bestTune$lambda)
To summarise, the optimal lambdas are given as:
0.055 by using
0.001 by using
I know that using
cv.glmnet() is not advisable, but I really want compare both methods using the same prerequisites. As main explanaition, I think the sampling approach for each fold might be an issue - but I use the same seeds and the results are quite different.
So I'm really stuck on why the two approaches are so different, while they should be quite similar? - I hope the community has some idea whats the issue here