Most Parsimonious Elastic Net Model - choosing $\alpha$ and $\lambda$

How do I calculate which Elastic Net model is the most regularized/parsimonious?

I am recreating GLMnet in another language as an exercise. I want to do a grid search over several values of alpha and lambda, and then take the most parsimonious model with a prediction error within 1 standard error of the mean of the smallest prediction error. (I think this is similar to what the R package caret does.)

My question is, how do I choose 'most parsimonious'? A higher lambda means stronger regularization, as does a higher alpha, but how do I combine the two?

• Optimize through cross validation, and all else equal, pick the model with the fewest non-zero features. That means less data to store and process.
– Emre
Commented Jun 15, 2015 at 18:00
• @Emre But what if I have two models with the same number of features but different coefficients? Commented Jun 16, 2015 at 14:55
• The same number of nonzero features AND cross validation metrics??
– Emre
Commented Jun 16, 2015 at 15:48
• Know that parimony is the enemy of predictive discrimination and stability. And what do you mean by prediction error? If it is not a proper accuracy scoring rule then model/feature selection on the basis of that measure will entail a ton of randomness. Also I'd like to see clear justification for the 1 SE rule. Commented Oct 2, 2015 at 12:46
• A higher alpha does not mean stronger regularization; alpha governs the balance between $L_1$ versus $L_2$ penalty rather than intensity of the penalty. Commented Oct 11, 2015 at 19:04

Imagine that the error surface is convex; the optimum is the point with the best out-of-sample performance metric. There is a contour line that corresponds to 1 s.e. around the maximum. Selecting any $(\alpha, \lambda)$ pair on that contour line which is within 1 s.e. of the optimum, but you have to decide upon a way to select among alternative $(\alpha, \lambda)$ pairs. One method would be to select the point on the contour which lies in the direction of larger $\lambda$ with the justification that no point on the contour has stronger regularization. This must be the most parsimonious model within 1 s.e. of the minimum because it applies the largest penalty to the coefficients.