2
$\begingroup$

How do I calculate which Elastic Net model is the most regularized/parsimonious?

I am recreating GLMnet in another language as an exercise. I want to do a grid search over several values of alpha and lambda, and then take the most parsimonious model with a prediction error within 1 standard error of the mean of the smallest prediction error. (I think this is similar to what the R package caret does.)

My question is, how do I choose 'most parsimonious'? A higher lambda means stronger regularization, as does a higher alpha, but how do I combine the two?

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ Optimize through cross validation, and all else equal, pick the model with the fewest non-zero features. That means less data to store and process. $\endgroup$
    – Emre
    Commented Jun 15, 2015 at 18:00
  • $\begingroup$ @Emre But what if I have two models with the same number of features but different coefficients? $\endgroup$
    – user5064
    Commented Jun 16, 2015 at 14:55
  • 1
    $\begingroup$ The same number of nonzero features AND cross validation metrics?? $\endgroup$
    – Emre
    Commented Jun 16, 2015 at 15:48
  • 2
    $\begingroup$ Know that parimony is the enemy of predictive discrimination and stability. And what do you mean by prediction error? If it is not a proper accuracy scoring rule then model/feature selection on the basis of that measure will entail a ton of randomness. Also I'd like to see clear justification for the 1 SE rule. $\endgroup$
    – Frank Harrell
    Commented Oct 2, 2015 at 12:46
  • 1
    $\begingroup$ A higher alpha does not mean stronger regularization; alpha governs the balance between $L_1$ versus $L_2$ penalty rather than intensity of the penalty. $\endgroup$
    – Richard Hardy
    Commented Oct 11, 2015 at 19:04

1 Answer 1

Reset to default
1
$\begingroup$

Imagine that the error surface is convex; the optimum is the point with the best out-of-sample performance metric. There is a contour line that corresponds to 1 s.e. around the maximum. Selecting any $(\alpha, \lambda)$ pair on that contour line which is within 1 s.e. of the optimum, but you have to decide upon a way to select among alternative $(\alpha, \lambda)$ pairs. One method would be to select the point on the contour which lies in the direction of larger $\lambda$ with the justification that no point on the contour has stronger regularization. This must be the most parsimonious model within 1 s.e. of the minimum because it applies the largest penalty to the coefficients.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.