I think when trying to interpret these plots of coefficients by $\lambda$, $\log(\lambda)$, or $\sum_i | \beta_i |$, it helps a lot to know how they look in some simple cases. In particular, how they look when your model design matrix is uncorrelated, vs. when there is correlation in your design.
To that end, I created some correlated and uncorrelated data to demonstrate:
x_uncorr <- matrix(runif(30000), nrow=10000)
y_uncorr <- 1 + 2*x_uncorr[,1] - x_uncorr[,2] + .5*x_uncorr[,3]
sigma <- matrix(c( 1, -.5, 0,
-.5, 1, -.5,
0, -.5, 1), nrow=3, byrow=TRUE
x_corr <- x_uncorr %*% sqrtm(sigma)
y_corr <- y_uncorr <- 1 + 2*x_corr[,1] - x_corr[,2] + .5*x_corr[,3]
x_uncorr has uncorrelated columns
> round(cor(x_uncorr), 2)
[,1] [,2] [,3]
[1,] 1.00 0.01 0.00
[2,] 0.01 1.00 -0.01
[3,] 0.00 -0.01 1.00
x_corr has a pre-set correlation between the columns
> round(cor(x_corr), 2)
[,1] [,2] [,3]
[1,] 1.00 -0.49 0.00
[2,] -0.49 1.00 -0.51
[3,] 0.00 -0.51 1.00
Now lets look at the lasso plots for both of these cases. First the uncorrelated data
gnet_uncorr <- glmnet(x_uncorr, y_uncorr)
A couple features stand out
- The predictors go into the model in the order of their magnitude of true linear regression coefficient.
- The coefficient path of each feature is a line (with respect to $\sum_i | \beta_i |$) is piecewise linear, and only changes when a new predictor enters the model. This is only true for the plot with respect to $\sum_i | \beta_i |$, and is a good reason to prefer it over the others.
- When a new predictor enters the model, it affects the slope of the coefficient path of all predictors already in the model in a deterministic way. For example, when the second predictor enters the model, the slope of the first coefficient path is cut in half. When the third predictor enters the model, the slope of the coefficient path is one third its original value.
These are all general facts that apply to lasso regression with uncorrelated data, and they can all be either proven by hand (good exercise!) or found in the literature.
Now lets do correlated data
gnet_corr <- glmnet(x_corr, y_corr)
You can read some things off this plot by comparing it to the uncorrelated case
- The first and second predictor paths have the same structure as the uncorrelated case until the third predictor enters the model, even though they are correlated. This is a special feature of the two predictor case, which I can explain in another answer if there is interest, it would take me a little far afield of the current discussion.
- On the other hand, once the third predictor enters the model, we see deviations from the picture we would expect if all three features were uncorrelated. The coefficient of the second feature flattens out, and the third feature rises to its final value. Notice the slope of the first feature is unaffected, which we would not anticipate if there were no correlations! Essentially, resources spend on the coefficients within a group of three or larger may be "traded around" until the assignment of minimal $\sum | \beta_i |$ is found.
So now let's look at your plot from the cars dataset and read some interesting things off (I reproduced your plot here so this discussion is easier to read):
A word of warning: I wrote the following analysis predicated on the assumption that the curves show the standardized coefficients, in this example they do not. Non-standardized coefficients are not dimensionless, and not comparable, so no conclusions may be drawn from them in terms of predictive importance. For the following analysis to be valid, please pretend that the plot is of the standardized coefficients, and please perform you own analysis on standardized coefficient paths.
- As you say, the
wt predictor seems very important. It enters the model first, and has a slow and steady descent to its final value. It does have a few correlations that make it a slightly bumpy ride,
am in particular seems to have a drastic effect when it enters.
am is also important. It comes in later, and is correlated with
wt, as it affects the slope of
wt in a violent way. It is also correlated with
qsec, because we do not see the predictable softening of slope when those enter. After these four variables have entered though, we do see the nice uncorrelated pattern, so it seems to be uncorrelated with all the predictors at the end.
- Something enters at around 2.25 on the x-axis, but its path itself is imperceptible, you can only detect it by its affect to the
cyl is quite facinating. It enters second, so is important for small models. After other variables, and especially
am enter, it is not so important anymore, and its trend reverses, eventually being all but removed. It seems like the effect of
cyl can be completely captured by the variables that enter at the end of the process. Whether it is more appropriate to use
cyl, or the complementary group of variables, really depends on the bias-variance tradeoff. Having the group in your final model would increase its variance significantly, but it may be the case that the lower bias makes up for it!
That's a small introduction to how I've learned to read information off of these plots. I think they are tons of fun!
Thanks for a great analysis. To report in simple terms, would you say that wt, am and cyl are 3 most important predictors of mpg. Also, if you want to create a model for prediction, which ones will you include based on this figure: wt, am and cyl ? Or some other combination. Also, you do not seem to need best lambda for analysis. Is it not important as in ridge regression?
I'd say the case for
am are clear cut, they are important.
cyl is much more subtle, it is important in a small model, but not at all relevant in a large one.
I would not be able to make a determination of what to include based only on the figure, that really must be answered the context of what you are doing. You could say that if you want a three predictor model, then
cyl are good choices, as they are relevant in the grand scheme of things, and should end up having reasonable effect sizes in a small model. This is predicated on the assumption that you have some external reason to desire a small three predictor model though.
It's true, this type of analysis looks over the entire spectrum of lambdas and lets you cull relationships over a range of model complexities. That said, for a final model, I think tuning an optimal lambda is very important. In the absence of other constraints, I would definitely use cross validation to find where along this spectrum the most predictive lambda is, and then use that lambda for a final model, and a final analysis.
The reason I recommend this has more to do with the right hand side of the graph than the left hand side. For some of the larger lambdas, it could be the case that the model is overfit to the training data. In this case, anything you deduced from the plot in this regime would be properties of the noise in the dataset instead of the structure of the statistical process itself. Once you have an estimate of the optimal $\lambda$, you have a sense for how much of the plot can be trusted.
In the other direction, sometimes there are outside constraints for how complex a model can be (implementation costs, legacy systems, explanatory minimalism, business interpretability, aesthetic patrimony) and this kind of inspection can really help you understand the shape of your data, and the tradeoffs you are making by choosing a smaller than optimal model.
glmnet(as.matrix(mtcars[-1]), mtcars[,1]). $\endgroup$
my_data_framereturns a data frame with one column, whereas
my_data_frame[, 1]both return a vector that is not "contained" by a data frame. Matrices, however, are actually just flat vectors with a special attribute that allows R to access them like a grid, so
my_matrix[1, 1], and
my_matrix[]will all return the first element of
my_matrix[, 1] returns the first column. $\endgroup$