3
$\begingroup$

This is quite coincidental as my question is nearly identical to this one asked shortly before, but I am also using elastic net regularization with R's glmnet library as a method of variable selection (but in my case it is for a Gaussian and not binomial family).

I have an instance where there does not appear to be any value of lambda which reduces the deviance of the fit. In the left figure below, the $\lambda$ values are selected automatically by cv.glmnet(); in the right figure below I have specified the values to be $\exp(\{-11\ldots-1\})$ (using 250 points evenly spaced between -11 and -1), but in either case it appears there the deviance or mean square error (MSE) is monotonically increasing with $\lambda$.

Does this say something useful about my system, and is there any value of $\lambda$ for which a meaningful interpretation of the selected variables can be extracted?

I would additionally appreciate reference to peer-reviewed literature or conference proceedings which discuss similar cases - I have come up empty-handed in my search.

Many thanks, community.

enter image description here

(For non-R users, the left dotted vertical line in each figure indicates the minimum MSE value - which in my problem corresponds to the minimum value of $\lambda$ selected in both cases - and the right dotted vertical line corresponds to the $\lambda$ of the minimum MSE + 1 standard error solution.)

$\endgroup$
3
$\begingroup$

The interpretation is exactly the same as I discussed in the Q & A you link to. All your right-hand plot has done is extend the penalty ($\log(\lambda)$) into a region of exponentially decreasing values, i.e. exponentially decreasing amounts of shrinkage.

The CV deviance of the model is flat over a range of values for $\log(\lambda)$ yet there is shrinkage being applied; sufficient shrinkage to remove some variables from the model completely. I'm not sure you can trust the values on the upper axis for your automatic plot - do you really have 1636 covariates in the model? - but for the left-hand plot the simplest/smaller model within 1 SE of the best model has somewhere between 100 and 87 covariates in it.

In your situation, the optimal shrinkage is the full model; but a model that does just as well as this is the 1 SE model. This model has some shrinkage applied, but it is not a large amount; there will still be many predictors with non-zero coefficients.

$\endgroup$
  • $\begingroup$ Sorry I did not interpret your other response carefully. In the other problem, the vertical lines at the center of the figure indicate that actually there is a detectible minimum but your main point that the CV is relatively flat is also true. It is rather difficult to accept what "relatively flat" is since that also depends on the y-axis scale, and this is determined by how much the MSE increases at the highest values of lambda selected. In all other examples I have seen some decrease in the MSE with a non-zero value of lambda and the analysis proceeds from there. $\endgroup$ – hatmatrix Oct 29 '14 at 3:39
  • 1
    $\begingroup$ @crippledlambda What would you expect if you had 100 independent/orthogonal predictors, all known to be strongly predictive of the response? A true solution would include all parameters and we would expect large (relatively) coefficients for each one. This would mean that the error should increase away from this full fit (similar to what you see but stronger). The true solution is dense; shrinkage methods like the lasso and the elastic net assume the solution is sparse. I'm not saying this is the case here, just that is one example of how you might see this sort of CV error trace. $\endgroup$ – Gavin Simpson Oct 29 '14 at 3:46
  • 1
    $\begingroup$ @crippledlambda then, in that case, you are seeing huge shrinkage for effectively no impact on the model's ability to explain variance in your sample. That is good! I think at low values of $\lambda$ you are overfitting so go with the 1SE mode fit. $\endgroup$ – Gavin Simpson Oct 29 '14 at 3:48
  • 1
    $\begingroup$ @crippledlambda your setting is sparse; you have remove ~ 1800 of the predictors from the model by shrinking their coefficients to zero. The CV is a stochastic procedure; samples are assigned to folds at random hence one would expect some variation between runs unless you set the random seed the same each time. If you want to hone in the best 1SE value of $\lambda$ you'll need to use a lot more points over range say -6 to -3. but really it isn't going to matter much in terms of prediction. $\endgroup$ – Gavin Simpson Oct 29 '14 at 4:11
  • 1
    $\begingroup$ Alternatively, run the CV quite a few times and record the $\lambda$ for the 1SE solution from each of these. Use a representative value for your chosen $\lambda$. $\endgroup$ – Gavin Simpson Oct 29 '14 at 4:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.