I was wondering if someone could provide some insight on the pros/cons of using built-in cross validation functions like cv.glmnet (https://www.rdocumentation.org/packages/glmnet/versions/3.0-2/topics/cv.glmnet), as opposed to constructing a manually coded cross validation scheme with caret.

In particular, I'm aiming to do a stratified cross-validation to ensure equivalent proportions of the response variable. I'm not sure whether this is possible with built in cv functions, but any insight would be much appreciated.


1 Answer 1


I think this is where it differs:

1.the loss function, or the metric used to decide on the best parameter is restricted to deviance in model (deviance), misclassification error or 1-Accuracy (class) and AUC 'AUC'. For caret, you can use those above and also kappa cohen, precision etc.

2.In terms of stratified cross-validation, this is not a real problem. You can generate the folds using caret and feed it in cv.glmnet:

data  = iris
dataFolds = createFolds(factor(data$Species),5)
fold_id = rep(1:length(dataFolds),sapply(dataFolds,length))

mdl1 = cv.glmnet(x=as.matrix(data[,1:4]),y=data[,5],alpha=1,
foldid = fold_id[order(unlist(dataFolds))],measure="class")

3.cv.glmnet will choose the lambda that is 1se from the lambda with the least error as the optimal lambda. See this post.

4.you cannot tweak vary alpha with cv.glmnet , meaning you will have to run cv.glmnet with multiple runs of alpha

5.speed. cv.glmnet runs faster than caret if you have a large dataset, because it does not store as much information as caret, for example:

fit_cv = function(){
foldid = fold_id[order(unlist(dataFolds))],measure="class")
fit_caret = function(){

Unit: nanoseconds
      expr min  lq  mean median  uq  max neval cld
    fit_cv 131 173 379.3  324.0 581  877    10   a
 fit_caret 132 263 550.1  440.5 587 1342    10   a

This will only increase as your dataset gets larger


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.