3
$\begingroup$

I tried the glmnet package to learn multi-response Gaussian family. I have looked at the coefficients of the final model. The result is odd. All the features have non-zero coefficients? How is it possible? I used the l1 norm (LASSO).

Before training, I had 20 features in my model, at the end also I have 20 features with non-zero coefficients!

mfit = cv.glmnet(x, y, family="mgaussian", alpha=1)
coef(mfit, s="lambda.min")

$y1
21 x 1 sparse Matrix of class "dgCMatrix"
                       1
(Intercept) -0.227938007
V1          -0.450036883
V2           0.557660650
V3           0.025960121
V4          -0.006926971
V5           0.938951150
V6           0.028106596
V7          -0.062906327
V8           0.042020881
V9           0.324555833
V10         -1.162720758
V11          1.392068904
V12          0.708822585
V13          0.138470220
V14         -0.361619604
V15          0.263752069
V16         -0.139336945
V17          0.020135397
V18         -0.086938292
V19          0.037916729
V20          0.004525174

$y2
21 x 1 sparse Matrix of class "dgCMatrix"
                       1
(Intercept) -0.152414042
V1           1.816832714
V2          -0.075907117
V3           0.233417492
V4          -0.542780903
V5          -0.038131010
V6          -0.033692294
V7           0.167815325
V8          -0.114406644
V9          -0.202872934
V10          0.023561811
V11          0.192387547
V12         -0.159011058
V13         -0.028944733
V14          0.484381888
V15         -0.009595264
V16         -0.070575757
V17         -0.158257184
V18         -0.636365334
V19         -0.393429761
V20         -0.179587606

$y3
21 x 1 sparse Matrix of class "dgCMatrix"
                       1
(Intercept)  0.064136138
V1           0.012343756
V2          -0.944652079
V3          -0.055213665
V4          -0.010847049
V5          -0.056957346
V6           0.732330922
V7          -0.016776548
V8          -0.580169864
V9          -0.328945770
V10          1.172705470
V11          0.019684395
V12          0.006571253
V13          0.143321523
V14         -0.017546337
V15         -0.306146331
V16         -0.282589578
V17          1.244944432
V18          0.028064436
V19          0.017680774
V20         -0.243873281

$y4
21 x 1 sparse Matrix of class "dgCMatrix"
                       1
(Intercept)  0.262237827
V1           1.173039843
V2          -0.084647045
V3          -0.070503854
V4           0.630234279
V5           0.021658875
V6          -0.068329527
V7           1.661538220
V8           0.708288249
V9           0.580157907
V10         -0.040516034
V11         -0.251500477
V12         -0.038651852
V13          0.279724140
V14         -0.091477066
V15         -0.557647544
V16         -0.046259431
V17         -1.265200899
V18         -0.008754935
V19          0.205192998
V20         -0.050390759
$\endgroup$
  • 1
    $\begingroup$ There's no rule that says LASSO has to be sparse, only that it can be $\endgroup$ – shadowtalker Jan 6 '15 at 16:10
  • $\begingroup$ come on, it's not make sense to have all features i your final model with L1 regularization. $\endgroup$ – user2806363 Jan 6 '15 at 16:56
  • 2
    $\begingroup$ that depends entirely on your data $\endgroup$ – shadowtalker Jan 6 '15 at 16:58
  • $\begingroup$ @user2806363 Think of a simple example: optimize (x - 1)² with L1-regularization lambda = 1. Of course this will not yield x=0 (it will yield x=0.5). So you see, if the L2 error is overpowering the L1 regularization, you will not get zero coefficients. $\endgroup$ – Owen Mar 29 '17 at 2:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.