7
$\begingroup$

Following plot is obtained on performing LASSO using glmnet package:

enter image description here

Is there any significance of area under the curves (using 0 as baseline) in reporting significance of the variables? Can we say that importance of different variables in predicting the dependent variable is reflected by the area under its curve? So here the purple and black variables are probably equally important. They are followed by (in descending order) deep blue, green, light blue and red variables.

$\endgroup$
  • $\begingroup$ The answer is probably yes. The point being that before applying shrinkage methods one should standardize the predictors so that shrinkage is indeed an indication of effect size. For more information see for example stats.stackexchange.com/questions/86434/… Issues of colinearity might make practical application difficult though. $\endgroup$ – spdrnl Jun 1 '15 at 11:59
  • $\begingroup$ Will ridge regression or elasticnet be better for comparison of coefficients to determine relative importance of variables? I believe, ridge regression is recommended in the setting of collinearity. $\endgroup$ – rnso Jun 1 '15 at 13:56
2
$\begingroup$

A couple things that immediately occur to me about this.

I think spdrnl's right, due to the standardization, the effect sizes should be comparable. It looks like it may be the case that the plot is on the scale of the original variables though, I'd check which is true and work with a plot of the coefficients of the standardized predictors.

First observation. I think you'll want to be careful with your region of integration. Suppose the most predictive model is associated with a $\log(\lambda)$ somewhere in the middle of the plot. Then the models corresponding to the left hand side of the plot are overfit, and just capturing noise in the data. You probably don't want to report on this area. So, in terms of lambda, I would recommend integrating:

$$ \int_0^{\lambda_{opt}} | \beta_i(t) | $$

Second observation. You are going to lose some subtlety with non-monotonic coefficient paths. I'm thinking of your lasso example from yesterday

enter image description here

Here the area method would report some definite significance for cyl. What's really true is that cyl is important for small models, then the effect drops out for large models. The area approach does not capture this. You may want to complement your area measurements with comments or pictures focusing on these interesting cases.

Finally, you'll have to choose what to measure on your x-axis. The choices are $\lambda$, $\log(\lambda)$ and $\sum_i | \beta_i |$. I would lean towards the latter, as that is measuring how much of the total allocated coefficient budget goes to each predictor. The others are only interpretable though Lagrange multipliers, making it hard to really be sure what is being measured.

$\endgroup$
  • $\begingroup$ How can we get sum of all betas which you suggest for each predictor for the range of x-axis from mod (mod = glmnet(as.matrix(mtcars[-1]), mtcars[,1]) )? $\endgroup$ – rnso Jun 1 '15 at 16:27
  • $\begingroup$ Try apply(abs(mod$beta), 2, sum). I'm not sure if mod$beta has the normalized or unnormalized coefficients though. $\endgroup$ – Matthew Drury Jun 1 '15 at 16:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.