1
$\begingroup$

This is an extension of the question on Variablity in cv.glmnet results. I've recently been running cross validation to find the best shrinkage parameter($\lambda$).

I was aware of the variability in cv.glmnet because it shuffles the training and validation data every time it gets called. So I called cv.glmnet multiple times in a for loop and created a histogram of frequencies of lambdas. The results were baffling:

enter image description here

On most iterations, this is the distribution I get. So the distribution of best lambdas is bimodal. But then after couple more iterations i get this: enter image description here

Despite the variability in cv.glmnet, I was at least hoping for a consistent distribution of lambdas. Am I doing something wrong in my code? Here is the code I wrote:

library(glmnet)
library(ISLR)
fix(Hitters)
Hitters = na.omit(Hitters)

x =model.matrix(Salary~., Hitters)[,-1]
y = Hitters$Salary

# Before Cross Validation, split data into train and test data
train=sample(1:nrow(x), nrow(x) * 0.8, replace=FALSE)
test=(-train)
y.test = y[test]

grid=10^seq(5, 2, length=200)
ridge.mod=glmnet(x[train,], y[train], alpha=0, lambda=grid, thresh=1e-12)

# Number of iteration of cross validation
n = 1000
mses = NULL
bestlams = NULL
minmses = NULL
for (i in 1:n)
{
  cv.out = cv.glmnet(x[train,], y[train], alpha=0, nfolds=10)
  mses = cv.out$cvm
  index = which.min(mses)
  minmses = cbind(minmses, cv.out$cvm[index])
  bestlams = cbind(bestlams, cv.out$lambda.min)
}

par(mfrow=c(2,2))
hist(log(minmses), 
     breaks=20,
     main="Min MSE Histogram",
     xlab="Log(Min MSE)",
     border="blue",
     col="green")
hist(bestlams, 
     breaks=20,
     main="Best lambda Histogram",
     xlab="Lambda",
     border="blue",
     col="green")

Here are some stuff I am thinking/thought about to fix this:

  1. change the number of iteration : changed it to n=1000, same results

  2. get rid of the intercept: this was an error on my end. I got rid of the intercepts in my feature matrix. But results were the same.

  3. specifying lambda parameter for cv.glmnet

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.