I am trying to perform the nested cross validation with glmnet and I want to tune both alpha and lambda. I want to pass the algorithm a sequence of possible alphas and let it decide for the lambda since very often I get the convergence issue if I supply the lambda seq. Anyhow, I found myself confused when it came to the inner loop and how to proceed. The (pseudo)code looks like this:

    #first, create folds for the original data:
    folds = createFolds(y=y, k=nr_outer_folds)
    for i from 1 to number of folds:
          #split into train, test (based on folds)
           x.train.outer = x[-folds[[i]],]
           x.test.outer = x[folds[[i]],]
           y.train.outer = y[-fold[i]]
           y.test.outer = y[fold[i]]

           #create folds for x.train.outer
           folds_inner = createFolds(y=y.train.outer, k=nr_inner_folds)

           for j from 1 to number of INNER folds:
              #split x.train.outer to x.train.inner, x.test.inner
              #split y.train.outer to y.train.inner, y.test.inner
                #Here is the place to tune the parameters
                 for each alpha from alpha_values
       test best parameters on x.test.outer and check the accuracy, save results
  • Now, if I proceed with cv.glmnet it is going to perform the cross-validation for given alpha in order to find suitable lambda. In my case, that would mean my data gets split once again which I don't want.
  • If I don't split x.train.outer to the inner manually, but go directly to the cv.glmnet with nfolds=nr_folds_inner, then I do not know what is my x.test.inner, i.e. where to test the model for different alphas and get the best parameters from inner loop which are then tested on x.test.outer?
  • If I use glmnet function, I get the sequence of lambdas, but how to choose one?

cv.glmnet documentation says:

If users would like to cross-validate alpha as well, they should call cv.glmnet with a pre-computed vector foldid, and then use this same fold vector in separate calls to cv.glmnet with different values of alpha.

Could somebody explain how exactly to use foldid in my case, or what should I do in general to properly use glmnet for nested cross validation?

Thank you!

  • $\begingroup$ The caret package implements a bunch of convenience functions around hyperparameter tuning for many machine learning algorithms, including glmnet. If all you need is to get this working, it could be easier to learn to use caret instead of rolling your own training loop. $\endgroup$ – Sycorax Dec 10 '18 at 23:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.