I am trying to perform the nested cross validation with glmnet and I want to tune both alpha and lambda. I want to pass the algorithm a sequence of possible alphas and let it decide for the lambda since very often I get the convergence issue if I supply the lambda seq. Anyhow, I found myself confused when it came to the inner loop and how to proceed. The (pseudo)code looks like this:
#first, create folds for the original data: folds = createFolds(y=y, k=nr_outer_folds) for i from 1 to number of folds: #split into train, test (based on folds) x.train.outer = x[-folds[[i]],] x.test.outer = x[folds[[i]],] y.train.outer = y[-fold[i]] y.test.outer = y[fold[i]] #create folds for x.train.outer folds_inner = createFolds(y=y.train.outer, k=nr_inner_folds) for j from 1 to number of INNER folds: #split x.train.outer to x.train.inner, x.test.inner #split y.train.outer to y.train.inner, y.test.inner .... #Here is the place to tune the parameters for each alpha from alpha_values ? test best parameters on x.test.outer and check the accuracy, save results
- Now, if I proceed with cv.glmnet it is going to perform the cross-validation for given alpha in order to find suitable lambda. In my case, that would mean my data gets split once again which I don't want.
- If I don't split x.train.outer to the inner manually, but go directly to the cv.glmnet with nfolds=nr_folds_inner, then I do not know what is my x.test.inner, i.e. where to test the model for different alphas and get the best parameters from inner loop which are then tested on x.test.outer?
- If I use glmnet function, I get the sequence of lambdas, but how to choose one?
cv.glmnet documentation says:
If users would like to cross-validate alpha as well, they should call cv.glmnet with a pre-computed vector foldid, and then use this same fold vector in separate calls to cv.glmnet with different values of alpha.
Could somebody explain how exactly to use foldid in my case, or what should I do in general to properly use glmnet for nested cross validation?