I am regressing a continuous predictor on over 60 variables (both continuous and categorical) using LASSO (glmnet).

In examining the variable trace plot, I notice that as log lambda increases, one of the key variables has a coefficient that actually increases. Then, after a certain point, it begins to decrease like we would expect.

To make sure this wasn't a fluke, I ran 10 models using bootstraps and obtained very similar results.

Is this possible, or is there a problem with the data? If legitimate, what does this trend in the variable's coefficient tell us about the variable and the relation to the response?

Initial variable trace plot 10 bootstrapped models


It's not only possible, it's a very common occurrence.

Note that the penalty is $\ \lambda\,||\beta||_1$. So some components can increase in magnitude as long as others decrease, without increasing the norm overall. Sometimes as $\lambda$ increases, one (or a few) coefficient(s) may increase in size at the expense of others which together decrease at least as rapidly, because it helps keep down the rate of increase in the lack of fit term more than reducing them all together would.

You might like to plot what happens to $\sum_i |\beta_i|$ as $\log\lambda$ increases.

You'll often see this kind of behaviour when there's some correlation amongst the predictors - there can be a sort of substitution effect.

Note that in your top plot $|\beta_4|+|\beta_{11}|$ is pretty nearly always decreasing or fairly stable (the occasional small increase will be offset by decreases in the coefficients of still other variables)

| cite | improve this answer | |
  • $\begingroup$ What does this dynamic say about the variable's effect on the response compared to if it were just decreasing with an increasing lambda? $\endgroup$ – matsuo_basho Apr 8 '16 at 3:42
  • $\begingroup$ I'm not exactly sure what you're asking there but you need to consider the effect of all the variables together. For example, if $\beta_4-\beta_{11}$ is nearly constant in a lot of that increase of $\beta_4$ with increasing $\lambda$, as we see in your top plot, you would probably want to consider what the effect of that contrast is. The effect of $x_4$ on its own may be puzzling because you're missing half the story there. $\endgroup$ – Glen_b Apr 8 '16 at 3:47
  • $\begingroup$ +1, but would it be possible to get a constructive example showing how and why such things happen? $\endgroup$ – Richard Hardy Apr 8 '16 at 12:23
  • $\begingroup$ Let me provide some background as to why I'm asking the question. I would like to identify the most important variables in the model. From the models I run, we see that variables 4 and either variable 11 or 24 are consistently in the model when lambda is high. Thus, we can say that they are important. Although variable 4 generally has a positive coefficient, the way it changes is a bit confusing. Does this dynamic tell us anything about interpreting the variables' effect on the response? $\endgroup$ – matsuo_basho Apr 8 '16 at 13:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.