Why does lambda.min value in glmnet tuning cross-validation change, when repeating test? I'm using glmnet package to build a linear regression with $\alpha$ = 0.5, to find best $\lambda$.
xMatrix [44x15000] ($p>>N$), y = numeric[44]
Everything seems to run ok, but $\lambda$ changes its value quite a lot when I re-run the command. Is my approach incorrect? 
cvGlmnet <- cv.glmnet(xMatrix, y, alpha=0.5, nfolds=10)  
 cvGlmnet$lambda.min  [1] 10.24038  
 cvGlmnet$lambda.1se  [1] 17.08198  
cvGlmnet <- cv.glmnet(xMatrix, y, alpha=0.5, nfolds=10)  
 cvGlmnet$lambda.min  [1] 14.85703  
 cvGlmnet$lambda.1se  [1] 17.08198  
cvGlmnet <- cv.glmnet(xMatrix, y, alpha=0.5, nfolds=10)  
 cvGlmnet$lambda.min  [1] 4.865008  
cvGlmnet <- cv.glmnet(xMatrix, y, alpha=0.5, nfolds=10)  
 cvGlmnet$lambda.min  [1] 16.30557
cvGlmnet <- cv.glmnet...  
 cvGlmnet$lambda.min  [1] 14.18176 

Besides this, I've also tried caretpackage to tune both $\alpha$ and $\lambda$. If I did it correct, obj$bestTune gave $\alpha$ = 1 and $\lambda$ = 3. When I try glmnetas above, but using $\alpha$ = 1 instead, $\lambda$ values move between 5-8 (at least what I've tried, they also vary between runnings). Shouldn't this give something ~3?
I've tried with this example
# generate a dummy dataset with 30 predictors (10 useful & 20 useless) 
y=rnorm(100)
x1=matrix(rnorm(100*20),100,20)
x2=matrix(y+rnorm(100*10),100,10)
x=cbind(x1,x2)

# use crossvalidation to find the best lambda
library(glmnet)
cv <- cv.glmnet(x,y,alpha=1,nfolds=10)
l <- cv$lambda.min

And it also changes slightly, but not as much as it does with my data. Is it because of my data structure/nature?
Using caret, also the alpha value changed quite a lot (considering that it can only range between 0 and 1 and each extrem is quite a dfferent approach). Any clue?
 A: The reason why you're getting different lambda values is because every time you call
cv.glmnet(xMatrix, y, alpha=0.5, nfolds=10), you're essentially creating different cross validation folds.To retain the same lambda value, you need to make sure you're using the same cross validation folds every time, thus you might want to try initializing a random number seed prior to invoking cv.glmnet(xMatrix, y, alpha=0.5, nfolds=10). Try running this whole chunk of code repeatedly:
set.seed(1)
cvGlmnet <- cv.glmnet(xMatrix, y, alpha=0.5, nfolds=10)
cvGlmnet$lambda.min 

You'll see that lambda remains the same. Hence, you'll have reproducible results! MichaelJ in the comments below your query essentially answered your question.
A: It is difficult to tell without the data to reproduce things. You didn't list the caret code so I'm not sure what was done there.
I think that the bottom line is that you have 44 samples and 10-fold CV, known to have high variance, is not going to give you repeatable results. I would suggest using several repeats of 10FCV (via trainControl's method = "repeatedcv" option) or go to the bootstrap and accept that your RMSE estimates are going to be a little pessimistic.
HTH,
Max
A: If you want to get repeatable optimal lambda & alpha, you can use leave-one-out CV (which doesn't support AUC though). 
cv <- cv.glmnet(x,y,alpha=1,nfolds=nrow(x))

It's pretty normal for an n-fold CV to bring you very huge variance, since the partitions of sample are generated randomly.
A: Your methodology is not great for reproducibility: you set.seed(1) once then run cv.glmnet() 100 times. Each of those calls to cv.glmnet() is itself calling sample() N times. So if the length of your data ever changes, the reproducibility changes.
Better to explicitly set.seed() right before each run. Or else keep the foldids constant across runs (use the utility functions from caret or ).
