I am working on a project where I am implementing Lasso regression in R for feature selection and my scenario is as follows.

For the minimum value of $\lambda$ most of the corresponding coefficients are zero (40 out 45 coefficients are zero). It is said that the coefficients will become zero when $\lambda$ is too high. On the contrary for me, the $\lambda$ value is very small (actually to the power of -5) i.e. I have a very small $\lambda$ value and most of the coefficients are zero.

So, I have a few questions listed below:

  1. Is this scenario common? Can I take any measures to prevent it?
  2. Is selecting by Lasso not suitable for feature selection in my scenario? If so, what are the other methods I can use?

Edit: Added R code below:

coef(cv.glmmod, s = "lambda.min")[which(coef(cv.glmmod, s = "lambda.min") != 0)]    
> 6.456279e-05 3.838600e-07 1.356334e-05
colnames(Final_raw)[which(coef(cv.glmmod, s = "lambda.min") != 0)]
> "smart_1_raw"   "smart_189_raw" "smart_198_raw"
plot(cv.glmmod) [![enter image description here][2]][2]
best_lambda <- cv.glmmod$lambda.min
> 9.175735e-05
(plot(cv.glmmod$glmnet.fit)) [![enter image description here][1]][1]

PS: I have also tried Ridge and Elastic net and the results were similar as the above.

Edit 2: After Up sampling: enter image description here enter image description here

  • $\begingroup$ Can you upload a plot of the coefficients versus lambda? plot(model) (plot(model$glmnet.fit) for CV'd models) gets you this in R, not sure about Python implementations. $\endgroup$ – Dex Groves Sep 26 '16 at 4:06
  • $\begingroup$ @Dex Groves. I have Uploaded the graphs and code. $\endgroup$ – Harini Devulapalli Sep 26 '16 at 10:48
  • $\begingroup$ These look very suspect. Your MAE seems low (though I don't know your target variable, so maybe it's reasonable). SE in error across folds is huge. Is your response highly skewed so that it is almost entirely one class? Is it possible that your covariates include something that is leaking, like a copy of the response variable? $\endgroup$ – Dex Groves Sep 26 '16 at 15:53
  • $\begingroup$ Apologies for late reply. As you said the response (Binary variable) in the data set is highly skewed. So I did up sampling to even the 0's and 1's in the data. I have uploaded the "plot(model)" , "(plot(model$glmnet.fit)" in the "edit 2" section above. Do you find these graphs better? $\endgroup$ – Harini Devulapalli Sep 29 '16 at 0:25
  • $\begingroup$ Those are much better, now you have 12 nonzero coefficients! If your response is binary, you should be setting family="binomial" for the glmnet call. Right now you're optimizing for squared error. I'm curious how changing the family would affect those graphs. $\endgroup$ – Dex Groves Sep 29 '16 at 0:45

The penalty parameter $\lambda$ must be chosen after your design matrix is scaled. From section 2.5 of the glmnet paper:

$$N\alpha\lambda_\max = \max_{l \in \{1, \ldots, p\}} \left\vert \sum_{i=1}^n X_{il} y_i \right\vert$$

Note $\alpha = 1$ for lasso regression. $\lambda_\max$ is the value of $\lambda$ for which all coefficients are 0. This function will find $\lambda_\max$ from your input design matrix $X$ and response vector $y$:

find_max_lambda <- function(X, y) {
   mysd <- function(x) sqrt(sum((x-mean(x))^2)/length(x))
   sX <- scale(X, scale = apply(X,2,mysd))
   max(abs(colSums(sX*y/length(y) ) ) )
  • $\begingroup$ The maximum lambda value i got is 0.0001541192 after implementing the above function. $\endgroup$ – Harini Devulapalli Sep 26 '16 at 11:05
  • 1
    $\begingroup$ suppose lambda_max = 1.54e-4 then you can try a regularization path of say 20 points using seq(exp(log(lambda_max), log(lambda_max * 1e-4), length.out = 20). By the time you reach the smallest value in this sequence, almost all your coefficients will be non-zero. $\endgroup$ – user3294195 Sep 26 '16 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.