Why would Lasso recommend a lambda of 0? I'm not too familiar with Lasso, but I did a grid search on my training data to determine lambda in a Lasso classification problem.  The Lambda that gives me the highest AUC is 0, which suggests that I don't shrink any of the coefficients.  This is hard to believe, because I know there are redundant variables in the data set - I just didn't want to go one by one and determine which ones to keep.  Was hoping Lasso would do it for me.
Any reason why Lambda is suggested to be 0?  When lambda is not 0, it gives me lower AUC than the base model, which also doesn't make sense.
Thanks!
 A: Note that model selection based on AUC is very noisy; I would not recommend it. The results you see may have little to do with differences in models. I'd change to deviance, which better reflects the differences in model fit.
It is possible that a very small lambda is what's needed. With enough data a zero coefficient can be found without penalty on the likelihood. If you by "redundant" simply mean correlated, which seems to be the case, it does not follow that the separate variables cannot both improve predictions. I attach a small simulation in R that shows how two highly correlated variables are both kept in by the lasso cross validation procedure since they both inform on the outcome.
Leaving either out will make predictions worse, the mse/lambda plot shows a clear worsening in error once the one variable starts to disappear. Farther below we also can see that even with a very small dataset it is possible that the second variable stays in.
If your goal is to get rid of some number of variables it is always possible to increase lambda until you have the desired number left, but I don't see why you'd want that if prediction alone is the goal.
A small simulation study:
library(glmnet)
#> Loading required package: Matrix
#> Loaded glmnet 4.1-1
library(MASS)

set.seed(20210305)
rho <- .8
R <- matrix(c(1, rho, rho, 1), nrow=2)

x <- MASS::mvrnorm(100, mu=c(0, 0), Sigma=R)

plot(x, main="x1 and x2 highly correlated")


# x1 and x2 both inform on y
y = 3 + 2*x[,1] - x[, 2] + rnorm(100, sd=.5)

# find a lambda by cross validation, alpha=1 is the lasso
cv_glmn <- cv.glmnet(x,y, alpha=1)

# best MSE reached with smallest lambda
plot(cv_glmn)



# even with only 10 observations, dropping a variable is not indicated
x <- MASS::mvrnorm(10, mu=c(0, 0), Sigma=R)
y = 3 + 2*x[,1] - x[, 2] + rnorm(10, sd=.5)
cv_glmn <- cv.glmnet(x,y, alpha=1)
#> Warning: Option grouped=FALSE enforced in cv.glmnet, since < 3 observations per
#> fold
plot(cv_glmn)


Created on 2021-03-05 by the reprex package (v0.3.0)
