4
$\begingroup$

I'm using glmnet package to build a linear regression with $\alpha$ = 0.5, to find best $\lambda$.
xMatrix [44x15000] ($p>>N$), y = numeric[44]
Everything seems to run ok, but $\lambda$ changes its value quite a lot when I re-run the command. Is my approach incorrect?

cvGlmnet <- cv.glmnet(xMatrix, y, alpha=0.5, nfolds=10)  
 cvGlmnet$lambda.min  [1] 10.24038  
 cvGlmnet$lambda.1se  [1] 17.08198  
cvGlmnet <- cv.glmnet(xMatrix, y, alpha=0.5, nfolds=10)  
 cvGlmnet$lambda.min  [1] 14.85703  
 cvGlmnet$lambda.1se  [1] 17.08198  
cvGlmnet <- cv.glmnet(xMatrix, y, alpha=0.5, nfolds=10)  
 cvGlmnet$lambda.min  [1] 4.865008  
cvGlmnet <- cv.glmnet(xMatrix, y, alpha=0.5, nfolds=10)  
 cvGlmnet$lambda.min  [1] 16.30557
cvGlmnet <- cv.glmnet...  
 cvGlmnet$lambda.min  [1] 14.18176 

Besides this, I've also tried caretpackage to tune both $\alpha$ and $\lambda$. If I did it correct, obj$bestTune gave $\alpha$ = 1 and $\lambda$ = 3. When I try glmnetas above, but using $\alpha$ = 1 instead, $\lambda$ values move between 5-8 (at least what I've tried, they also vary between runnings). Shouldn't this give something ~3?

I've tried with this example

# generate a dummy dataset with 30 predictors (10 useful & 20 useless) 
y=rnorm(100)
x1=matrix(rnorm(100*20),100,20)
x2=matrix(y+rnorm(100*10),100,10)
x=cbind(x1,x2)

# use crossvalidation to find the best lambda
library(glmnet)
cv <- cv.glmnet(x,y,alpha=1,nfolds=10)
l <- cv$lambda.min

And it also changes slightly, but not as much as it does with my data. Is it because of my data structure/nature?

Using caret, also the alpha value changed quite a lot (considering that it can only range between 0 and 1 and each extrem is quite a dfferent approach). Any clue?

$\endgroup$
3
$\begingroup$

It is difficult to tell without the data to reproduce things. You didn't list the caret code so I'm not sure what was done there.

I think that the bottom line is that you have 44 samples and 10-fold CV, known to have high variance, is not going to give you repeatable results. I would suggest using several repeats of 10FCV (via trainControl's method = "repeatedcv" option) or go to the bootstrap and accept that your RMSE estimates are going to be a little pessimistic.

HTH,

Max

$\endgroup$
  • $\begingroup$ The reduced num of obs. (~4/fold) can lead to great variability between folds (and this can lead to overall bad performances, ok) but I think that the lambda obtained shouldn't change just because of running the same command, say, 5 minutes later. What I mean is that everytime I execute cvGlmnet <- cv.glmnet(xMatrix, y, alpha=0.5, nfolds=10) the composition of each fold should be the same, shouldn't be? $\endgroup$ – PGreen Sep 27 '13 at 18:21
  • 3
    $\begingroup$ No. The folds are generated by random sampling and the state of the random number generator changes after each run. If you want reproducible results you should seed the random generator to a specific value beforehand using, for example, set.seed(1). $\endgroup$ – MichaelJ Jun 12 '14 at 4:07
6
$\begingroup$

The reason why you're getting different lambda values is because every time you call cv.glmnet(xMatrix, y, alpha=0.5, nfolds=10), you're essentially creating different cross validation folds.To retain the same lambda value, you need to make sure you're using the same cross validation folds every time, thus you might want to try initializing a random number seed prior to invoking cv.glmnet(xMatrix, y, alpha=0.5, nfolds=10). Try running this whole chunk of code repeatedly:

set.seed(1)
cvGlmnet <- cv.glmnet(xMatrix, y, alpha=0.5, nfolds=10)
cvGlmnet$lambda.min 

You'll see that lambda remains the same. Hence, you'll have reproducible results! MichaelJ in the comments below your query essentially answered your question.

$\endgroup$
4
$\begingroup$

If you want to get repeatable optimal lambda & alpha, you can use leave-one-out CV (which doesn't support AUC though).

cv <- cv.glmnet(x,y,alpha=1,nfolds=nrow(x))

It's pretty normal for an n-fold CV to bring you very huge variance, since the partitions of sample are generated randomly.

$\endgroup$
0
$\begingroup$

Your methodology is not great for reproducibility: you set.seed(1) once then run cv.glmnet() 100 times. Each of those calls to cv.glmnet() is itself calling sample() N times. So if the length of your data ever changes, the reproducibility changes.

Better to explicitly set.seed() right before each run. Or else keep the foldids constant across runs (use the utility functions from caret or ).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.