3
$\begingroup$

I'm trying to use glmnet packaget to perform elastic net model for binomial regression. I got the following graph: enter image description here

What I see from this is that increasing the lambda never decreases the deviance, so does this mean that no penalization is better that any penalization over $L_1$ & $L_2$?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
5
$\begingroup$

I'm not sure why you say that the deviance never decreases; it is certainly decreasing at large values of $\log(\lambda)$ on the rightmost edge of the plot to the left. However, the deviance is relatively flat over a large range of values of $\log(\lambda)$. Yet shrinkage is being applied, as the values on the upper x-axis (side = 3) indicate: the number of variables in the model declines consistently and markedly over the region of no or little change in deviance.

What this shows is that for models over this region, up to about $\log(\lambda) = -6$, shrinkage is being applied without affecting the CV fit of the model as assessed via CV binomial deviance. This is good, as it indicates that the elastic net is doing what you wanted it to, to perform feature selection whilst handling correlated covariates.

Improve this answer
$\endgroup$
3
  • $\begingroup$ But the log lambda is decreasing going to the left, so it means that lambda (that can be between 0 and 1) goes down to zero. What I'm trying to say is that the curve never bends (2nd derivative never changes sign) so if everytime I reduce lambda I get less error, in this model what is the purpose of the penalization part? $\endgroup$
    – GabyLP
    Commented Oct 29, 2014 at 3:00
  • $\begingroup$ @GabyP You are mixing up $\alpha$ and $\lambda$. The former, $\alpha$ is the mixing of L1 and L2 penalties, and is set as argument alpha in the cv.glmnet call. $\lambda$ is the amount of shrinkage applied. cv.glmnet optimises over $\lambda$ only. You need to do your own procedure if you wish to tune over both $\lambda$ and $\alpha$. If this is what you were trying to do, consider the caret package, which should be able to tune over both parameters for you using glmnet models. $\endgroup$
    – Gavin Simpson
    Commented Oct 29, 2014 at 3:05
  • 1
    $\begingroup$ The penalization is in terms of shrinkage of model coefficients. On the left you have the CV binomial deviance for the full, unpenalized model; on the right you have a heavily shrunk fit with large penalties. What this plot is showing you is that you can shrink the coefficients of a lot covariates by an amount sufficient to remove them from the model (their coefficient becomes zero) without affecting the model fit - the deviance of the model doesn't change much. Only at large penalties (large values of $\log(\lambda)$) does the model start to loose fit relative to the full model. $\endgroup$
    – Gavin Simpson
    Commented Oct 29, 2014 at 3:12
0
$\begingroup$

If you look at the gray bar that goes through the red dot(deviance), it indicates the range of variation in deviance after you do cross validation. The left side of chart with small penalty almost always have large range for deviance variation, meaning the model with less penalty is not stable enough. This is because, with a lot more independent variables, the model is capturing too much noise that comes with those less important variables, the model is overfitted. But on right side, the bar is small because with large penalty, the model almost always goes to the one strong driver, hence the model is more stable. So picking the right penalty is dealing with the trade-off between small deviance and stability of model. Therefore, adding the right penalty is a good thing!

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.