This is an extension of the question on Variablity in cv.glmnet results. I've recently been running cross validation to find the best shrinkage parameter($\lambda$).
I was aware of the variability in cv.glmnet because it shuffles the training and validation data every time it gets called. So I called cv.glmnet multiple times in a for loop and created a histogram of frequencies of lambdas. The results were baffling:
On most iterations, this is the distribution I get. So the distribution of best lambdas is bimodal. But then after couple more iterations i get this:
Despite the variability in cv.glmnet, I was at least hoping for a consistent distribution of lambdas. Am I doing something wrong in my code? Here is the code I wrote:
library(glmnet)
library(ISLR)
fix(Hitters)
Hitters = na.omit(Hitters)
x =model.matrix(Salary~., Hitters)[,-1]
y = Hitters$Salary
# Before Cross Validation, split data into train and test data
train=sample(1:nrow(x), nrow(x) * 0.8, replace=FALSE)
test=(-train)
y.test = y[test]
grid=10^seq(5, 2, length=200)
ridge.mod=glmnet(x[train,], y[train], alpha=0, lambda=grid, thresh=1e-12)
# Number of iteration of cross validation
n = 1000
mses = NULL
bestlams = NULL
minmses = NULL
for (i in 1:n)
{
cv.out = cv.glmnet(x[train,], y[train], alpha=0, nfolds=10)
mses = cv.out$cvm
index = which.min(mses)
minmses = cbind(minmses, cv.out$cvm[index])
bestlams = cbind(bestlams, cv.out$lambda.min)
}
par(mfrow=c(2,2))
hist(log(minmses),
breaks=20,
main="Min MSE Histogram",
xlab="Log(Min MSE)",
border="blue",
col="green")
hist(bestlams,
breaks=20,
main="Best lambda Histogram",
xlab="Lambda",
border="blue",
col="green")
Here are some stuff I am thinking/thought about to fix this:
change the number of iteration: changed it to n=1000, same resultsget rid of the intercept: this was an error on my end. I got rid of the intercepts in my feature matrix. But results were the same.specifying lambda parameter for cv.glmnet