# Cross validation with two parameters: elastic net case

I want to know the cross validation procedure to find the two parameters of elastic net presented by Zou and Hastie on the prostate dataset as example. I can't improve the error rate lasso with k-fold when I use elastic net.

• Thanks for the answers. In fact for one parameters i well know the procedure but for two it is not clear. Do you have an example like elastic net and prostate data? i try to find the results of the paper i mentioned. Any references for simplex use ? Thanks – user7145 Oct 31 '11 at 23:11
• Do you have a R example on prostate data like in the paper i mentionned ? – grant Nov 2 '11 at 18:09

The method to use in this case is exactly the same, though e.g. the glmnet package doesn't provide it out of the box.

Instead of working over 1 discrete set of parameter values (lambda), you now crossvalidate for a grid of parameter values, (lambda and alpha), then pick the best value (lambda.min and alpha.min), and then the lambda and alpha so that lambda is the biggest possible but its predictive measure is within 1 SE of that of lambda.min and alpha.min.

If you use R, you can probably do something like:

alphasOfInterest<-seq(0,1,by=0.1) #or something similar
#step 1: do all crossvalidations for each alpha
cvs<-lapply(alphasOfInterest, function(curAlpha){
cv.glmnet(myX, myY, alpha=curAlpha, some more parameters)
})
#step 2: collect the optimum lambda for each alpha
optimumPerAlpha<-sapply(seq_along(alphasOfInterest), function(curi){
curcvs<-cvs[[curi]]
curAlpha<-alphasOfInterest[curi]
indOfMin<-match(curcvs$lambda.min, curcvs$lambda)
c(lam=curcvs$lambda.min, alph=curAlpha, cvup=curcvs$cvup[indOfMin])
})
#step 3: find the overall optimum
posOfOptimum<-which.min(optimumPerAlpha["lam",])
overall.lambda.min<-optimumPerAlpha["lam",posOfOptimum]
overall.alpha.min<-optimumPerAlpha["alph",posOfOptimum]
overall.criterionthreshold<-optimumPerAlpha["cvup",posOfOptimum]
#step 4: now check for each alpha which lambda is the best within the threshold
corrected1se<-sapply(seq_along(alphasOfInterest), function(curi){
curcvs<-cvs[[curi]]
lams<-curcvs$lambda lams[lams<overall.lambda.min]<-NA lams[curcvs$cvm > overall.criterionthreshold]<-NA
lam1se<-max(lams, na.rm=TRUE)
c(lam=lam1se, alph=alphasOfInterest[curi])
})
#step 5: find the best (lowest) of these lambdas
overall.lambda.1se<-max(corrected1se["lam", ])
pos<-match(overall.lambda.1se, corrected1se["lam", ])
overall.alpha.&se<-corrected1se["alph", pos]


All this code is untested + needs attention if you use auc as your criterion (because then you need to look for the maximum of the criterion and some other details change), but the ideas are there.

Note: in the last step, you could, instead of going for the highest lambda, find the one that has the most parsimonious model (because higher lambda does not guarantee more parsimony over different alphas)

You may also want to collect all lambdas up front, and pass the collection of all those to every crossvalidation, so that you can ensure that each crossvalidation uses the same set of lambdas. This is easy to do but requires some extra steps. I'm not certain whether it is necessary...

• tahnks for the answer. I am not surety understand the procedure. 1) and 2) get the grid of crosss validation error 3) find the best pair of parameters ? i don't know the step 4 and 5 thanks – user7114 Oct 29 '11 at 10:16
• Just to clarify, you're 1) doing CVs for each alpha/lambda pair, 2) associating the lowest lambda of each alpha with it's alpha, 3) finding the lowest lambda of the overall (alpha, lowest lambda) set, 4) creating a new set which includes only CVs in the (alpha, lowest lambda) set with a higher lambda, lower error, and within 1 SE of the 'overall optimum', and 5) choosing the (alpha, lambda) pair from that subset with the highest lambda. – Peter Kazazes Mar 22 '16 at 22:53
• I'm not sure how glmnet eliminates lambdas, but wouldn't the lowest lambda in step 2 always be your lowest lambda value from the CV? – Peter Kazazes Mar 22 '16 at 22:55
• Thank you for your very helpful post @ Nick Sabbe. I tried to compare your way of selecting alpha and lambda with the caretpackage. Surprisingly, both ways result in very different estimations of alpha and lambda. Do you have an idea why this could be the case? In this thread you can find my (reproducible) code concerning the comparison of both methods: stats.stackexchange.com/questions/268885/… – Joachim Schork Mar 23 '17 at 10:14

If by error rate you mean the usual one (proportion classified correctly), this is a discontinuous improper scoring rule. An improper scoring rule is optimized by a bogus model. It will lead you to select the wrong features and to discard features that are predictive. It is better to use the most sensitive measure for assessing a model, which will be based on deviance or penalized deviance. In simpler cases where I'm only using a quadratic penalty, I solve for the optimum penalty using effective AIC. An example is in my book Regression Modeling Strategies. A simulation study of this approach is on http://biostat.mc.vanderbilt.edu/rms.

Rather than use a grid search, use a numeric optimisation routine, such as the Nelder-Mead simplex algorithm. This is generally more efficient than grid search and it will be well worth the effort in the long run. In MATLAB this is implemented by the fminsearch routine of the optimisation toolbox, but I expect there is an R implementation as well. The cost function for the optimisation is simply the cross-validated performance estimate.

It is a good idea to re-parameterise the problem first so as to achieve a non-constrained optimisation problem (the regularisation parameters must be positive). I do this by optimising the logarithm of the regularisation parameters, and have found it generally works well.