24
$\begingroup$

Following is the plot of glmnet with default alpha(1, hence lasso) using mtcars data set in R with mpg as the DV and others as the predictor variables.

glmnet(as.matrix(mtcars[-1]), mtcars[,1])

enter image description here

What can we conclude from this plot regarding different variables, especially am, cyl and wt (red, black and light blue lines)? How would we phrase the output in a report to be published?

I thought of following:

  1. wt is the most important predictor of mpg. It is negatively affecting the mpg.

  2. cyl is a weak negative predictor of mpg.

  3. am may be a positive predictor of mpg.

  4. Other variables are not robust predictors of mpg.

Thanks for your thoughts on this.

(Note: cyl is the black line which does not reach 0 till very close to it.)

Edit: Following is plot(mod, xvar='lambda') which shows x-axis in order reverse of above plot:

enter image description here

$\endgroup$
9
  • $\begingroup$ It looks like you forgot a comma before -1 in glmnet(as.matrix(mtcars[-1]), mtcars[,1]). $\endgroup$ Commented May 31, 2015 at 12:25
  • $\begingroup$ If comma is not given, R assumes the number to be column number, so it works. $\endgroup$
    – rnso
    Commented May 31, 2015 at 12:26
  • $\begingroup$ Nice, I did not now that before. $\endgroup$ Commented May 31, 2015 at 12:30
  • 2
    $\begingroup$ @RichardHardy be careful; this behavior is different for data frames and matrices. A data frame is a list, and each column is an element of that list, so my_data_frame[1] returns a data frame with one column, whereas my_data_frame[[1]] and my_data_frame[, 1] both return a vector that is not "contained" by a data frame. Matrices, however, are actually just flat vectors with a special attribute that allows R to access them like a grid, so my_matrix[1], my_matrix[1, 1], and my_matrix[[1]] will all return the first element of my_matrix. my_matrix[, 1] returns the first column. $\endgroup$ Commented May 31, 2015 at 14:03
  • 2
    $\begingroup$ I just mention that the plot_glmnet function in the plotmo package gives slightly better coefficient plots because it takes care of label overplotting and some other details. Examples may be found in Chapter 6 in plotres vignette. $\endgroup$ Commented Nov 29, 2016 at 22:23

1 Answer 1

49
+50
$\begingroup$

I think when trying to interpret these plots of coefficients by $\lambda$, $\log(\lambda)$, or $\sum_i | \beta_i |$, it helps a lot to know how they look in some simple cases. In particular, how they look when your model design matrix is uncorrelated, vs. when there is correlation in your design.

To that end, I created some correlated and uncorrelated data to demonstrate:

x_uncorr <- matrix(runif(30000), nrow=10000)
y_uncorr <- 1 + 2*x_uncorr[,1] - x_uncorr[,2] + .5*x_uncorr[,3]

sigma <- matrix(c(  1, -.5,   0,
                  -.5,   1, -.5,
                    0, -.5,   1), nrow=3, byrow=TRUE
)
x_corr <- x_uncorr %*% sqrtm(sigma)
y_corr <- y_uncorr <- 1 + 2*x_corr[,1] - x_corr[,2] + .5*x_corr[,3]

The data x_uncorr has uncorrelated columns

> round(cor(x_uncorr), 2)
     [,1]  [,2]  [,3]
[1,] 1.00  0.01  0.00
[2,] 0.01  1.00 -0.01
[3,] 0.00 -0.01  1.00

while x_corr has a pre-set correlation between the columns

> round(cor(x_corr), 2)
      [,1]  [,2]  [,3]
[1,]  1.00 -0.49  0.00
[2,] -0.49  1.00 -0.51
[3,]  0.00 -0.51  1.00

Now lets look at the lasso plots for both of these cases. First the uncorrelated data

gnet_uncorr <- glmnet(x_uncorr, y_uncorr)
plot(gnet_uncorr)

enter image description here

A couple features stand out

  • The predictors go into the model in the order of their magnitude of true linear regression coefficient.
  • The coefficient path of each feature is a line (with respect to $\sum_i | \beta_i |$) is piecewise linear, and only changes when a new predictor enters the model. This is only true for the plot with respect to $\sum_i | \beta_i |$, and is a good reason to prefer it over the others.
  • When a new predictor enters the model, it affects the slope of the coefficient path of all predictors already in the model in a deterministic way. For example, when the second predictor enters the model, the slope of the first coefficient path is cut in half. When the third predictor enters the model, the slope of the coefficient path is one third its original value.

These are all general facts that apply to lasso regression with uncorrelated data, and they can all be either proven by hand (good exercise!) or found in the literature.

Now lets do correlated data

gnet_corr <- glmnet(x_corr, y_corr)
plot(gnet_corr)

enter image description here

You can read some things off this plot by comparing it to the uncorrelated case

  • The first and second predictor paths have the same structure as the uncorrelated case until the third predictor enters the model, even though they are correlated. This is a special feature of the two predictor case, which I can explain in another answer if there is interest, it would take me a little far afield of the current discussion.
  • On the other hand, once the third predictor enters the model, we see deviations from the picture we would expect if all three features were uncorrelated. The coefficient of the second feature flattens out, and the third feature rises to its final value. Notice the slope of the first feature is unaffected, which we would not anticipate if there were no correlations! Essentially, resources spend on the coefficients within a group of three or larger may be "traded around" until the assignment of minimal $\sum | \beta_i |$ is found.

So now let's look at your plot from the cars dataset and read some interesting things off (I reproduced your plot here so this discussion is easier to read):

A word of warning: I wrote the following analysis predicated on the assumption that the curves show the standardized coefficients, in this example they do not. Non-standardized coefficients are not dimensionless, and not comparable, so no conclusions may be drawn from them in terms of predictive importance. For the following analysis to be valid, please pretend that the plot is of the standardized coefficients, and please perform you own analysis on standardized coefficient paths.

enter image description here

  • As you say, the wt predictor seems very important. It enters the model first, and has a slow and steady descent to its final value. It does have a few correlations that make it a slightly bumpy ride, am in particular seems to have a drastic effect when it enters.
  • am is also important. It comes in later, and is correlated with wt, as it affects the slope of wt in a violent way. It is also correlated with carb and qsec, because we do not see the predictable softening of slope when those enter. After these four variables have entered though, we do see the nice uncorrelated pattern, so it seems to be uncorrelated with all the predictors at the end.
  • Something enters at around 2.25 on the x-axis, but its path itself is imperceptible, you can only detect it by its affect to the cyl and wt parameters.
  • cyl is quite facinating. It enters second, so is important for small models. After other variables, and especially am enter, it is not so important anymore, and its trend reverses, eventually being all but removed. It seems like the effect of cyl can be completely captured by the variables that enter at the end of the process. Whether it is more appropriate to use cyl, or the complementary group of variables, really depends on the bias-variance tradeoff. Having the group in your final model would increase its variance significantly, but it may be the case that the lower bias makes up for it!

That's a small introduction to how I've learned to read information off of these plots. I think they are tons of fun!

Thanks for a great analysis. To report in simple terms, would you say that wt, am and cyl are 3 most important predictors of mpg. Also, if you want to create a model for prediction, which ones will you include based on this figure: wt, am and cyl ? Or some other combination. Also, you do not seem to need best lambda for analysis. Is it not important as in ridge regression?

I'd say the case for wt and am are clear cut, they are important. cyl is much more subtle, it is important in a small model, but not at all relevant in a large one.

I would not be able to make a determination of what to include based only on the figure, that really must be answered the context of what you are doing. You could say that if you want a three predictor model, then wt, am and cyl are good choices, as they are relevant in the grand scheme of things, and should end up having reasonable effect sizes in a small model. This is predicated on the assumption that you have some external reason to desire a small three predictor model though.

It's true, this type of analysis looks over the entire spectrum of lambdas and lets you cull relationships over a range of model complexities. That said, for a final model, I think tuning an optimal lambda is very important. In the absence of other constraints, I would definitely use cross validation to find where along this spectrum the most predictive lambda is, and then use that lambda for a final model, and a final analysis.

The reason I recommend this has more to do with the right hand side of the graph than the left hand side. For some of the larger lambdas, it could be the case that the model is overfit to the training data. In this case, anything you deduced from the plot in this regime would be properties of the noise in the dataset instead of the structure of the statistical process itself. Once you have an estimate of the optimal $\lambda$, you have a sense for how much of the plot can be trusted.

In the other direction, sometimes there are outside constraints for how complex a model can be (implementation costs, legacy systems, explanatory minimalism, business interpretability, aesthetic patrimony) and this kind of inspection can really help you understand the shape of your data, and the tradeoffs you are making by choosing a smaller than optimal model.

$\endgroup$
7
  • $\begingroup$ Thanks for a great analysis. To report in simple terms, would you say that wt, am and cyl are 3 most important predictors of mpg. Also, if you want to create a model for prediction, which ones will you include based on this figure: wt, am and cyl ? Or some other combination. Also, you do not seem to need best lambda for analysis. Is it not important as in ridge regression? $\endgroup$
    – rnso
    Commented May 31, 2015 at 18:13
  • $\begingroup$ @rnso I appended a response to my answer. $\endgroup$ Commented May 31, 2015 at 18:23
  • $\begingroup$ Perhaps 'patrimony' is supposed to be 'parsimony'? $\endgroup$
    – Hatshepsut
    Commented Apr 20, 2016 at 5:17
  • 1
    $\begingroup$ 1) I really meant "when the second predictor that enters the model enters the model", nothing to do with how the predictors are arranged in a data table. 2) One of the things I am trying to express here is the great inherit complexity in a question like that. There is no clear cut way to say "this predictor is important" and "this predictor is not important", or really even any consistent way to measure the importance of a predictor. The world just is not that simple. 3) Cross validation or out of sample testing can help you choose a most predictive model. $\endgroup$ Commented Jul 4, 2016 at 5:04
  • 5
    $\begingroup$ +6, I evidently missed this when it came out. Really nice work. $\endgroup$ Commented Jul 19, 2016 at 22:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.