1
$\begingroup$

Edited Question, since it was a duplicate I used Matlab to make a lasso model for my data that has 41 predictors and 1 response variable, and perhaps I used more variables that I need too or maybe some variables are not meaningful since some regression coefficients are 0.

For the non 0 coefficients, I got some that are 0.2, 0.8, 2.7, etc.

I understood that I can interpret that the higher the regression coefficient higher the importance for that respective variable, is there a Metric or a Rule of thumb that say if a regression coefficient is 10/50/100 times lower than the highest regression coefficient we can "reject" or "not consider" that variable, when implementing the model online? or since it gave a non zero value I should stick with that variable no matter what?

|cite|improve this question
$\endgroup$
  • 2
    $\begingroup$ This issue of feature importance is tricky and is discussed extensively on this site. This page has extensive discussion and links to further discussion. As does this page. Please look over those pages and their links and edit your question to specify any specific statistical issues that are still unclear to you. Also, please edit your question to spell out the meanings of "RR" and "EN" as not all readers will know what you mean by them. $\endgroup$ – EdM Apr 16 at 14:59
  • 1
    $\begingroup$ I know of no such rule. $\endgroup$ – Peter Flom - Reinstate Monica♦ Apr 17 at 11:31
  • $\begingroup$ @PeterFlom so if it gave a coefficient > 0, I should stick with that variable, no matter the value? $\endgroup$ – Tiago Dias Apr 17 at 13:37
  • 1
    $\begingroup$ It's not good, in statistics, to make universal rules for model building. You have to think about what you are doing. But you shouldn't reject a variable just because it is non-significant or has a low value. Model building requires thought. $\endgroup$ – Peter Flom - Reinstate Monica♦ Apr 18 at 11:53
1
$\begingroup$

If you want to assess the importance of features in the lasso framework, you can use stability selection by Meinshausen/Bühlmann. This means basically that you repeat your lasso $B$ times on a random subset of your data and in every run you check which features are in the top $L$ chosen features. In the end you give a score to every feature how often it was selected in the top $L$ over $B$ runs. The cited paper shows that stability selection is much more stable than simple lasso.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Yes I run lasso 30 runs, and for each run, I order the absolute value of the regression in a ranking of 1 to 41 (41 predictors) to map it, but I was not certain if it is a good methodology. Not sure if I can perform the calculation of the p-value for the regression coefficients for LAsso $\endgroup$ – Tiago Dias Apr 17 at 13:53
  • 1
    $\begingroup$ $B=30$ is probably too low, $B=100$ or even $=500$ should be better. The size of the subsample should be around half of the observations, if you don't have many observations you can choose to subsample some more. $L=5$ is the most common choice, this makes lasso run quite fast. $\endgroup$ – Edgar Apr 17 at 14:50
  • $\begingroup$ You don't have $p$-values this way, but the stability score should give you a good measure of how important the features are (Meinshausen and Bühlmann say that values above 0.6 are good indicators). Feel free to mark the answer of your choice as a solution to your problem! $\endgroup$ – Edgar Apr 17 at 14:52
  • $\begingroup$ BTW you should definitely not order by the size of the regression coefficients (as @EdM pointed out, too). You should order by the entrance of features into the model with Lasso (Lasso adds features one by one, with stability selection, you stop after $L=5$ and go on to the next run). $\endgroup$ – Edgar Apr 17 at 15:00
  • 1
    $\begingroup$ Maybe you should intensify your understanding of lasso first before you study more complicated methods that improve on lasso. $\endgroup$ – Edgar Apr 18 at 11:01
1
$\begingroup$

You have to think carefully about "importance" of selected predictors and what "p-values" really mean in LASSO.

Predictor importance

Demonstrations of LASSO can be based on a simulated data set with a small number of predictors associated with outcome and a large number that are not. In that context it works well to find the truly important predictors.

But in real-world applications, with multiple predictors that are correlated with each other, the choice of "important" predictors will vary from sample to sample from the same population. The variability in "importance" among predictors you saw among re-samples of your data, and that forms the basis of the stability selection method recommended in the answer by @Edgar, should lead to some questions about what "importance" of individual predictors means when there are multiple correlated predictors related to outcome.

Even when LASSO returns a value of 0 for a predictor's coefficient (as it is designed to do), that doesn't mean it's "not meaningful"; it just means that it didn't add enough to the model to matter for your particular sample and sample size. The predictors that were selected might be important within your particular data sample, but that doesn't mean they are the most important in any fundamental sense in the overall population and they certainly can't be interpreted to have causal effects on outcome.

Your particular approach based on ranking of coefficient values is potentially dangerous, depending on how it is done. Predictors are typically standardized before LASSO so that differences in measurement scales don't differentially affect the penalization of the coefficients. But some software then re-scales the coefficients to the original measurement scales. So at the least you have to be careful about whether you are ranking coefficients for standardized or for re-scaled predictors. You don't want the importance of a predictor having a length value to differ depending on whether you measured it in millimeters or miles.

LASSO p-values

In many applications, the most important issue with LASSO is how well the model works for prediction. A strength of LASSO is that, even with its potentially unstable selection among correlated predictors, models can work quite well in practice for prediction. In that context, p-values for individual coefficients are of little interest.

It's when you are interested in inference that p-values matter. This is a very difficult problem in LASSO or in any modeling approach that uses outcomes to select predictors. The usual assumptions for estimating p-values in standard regression models no longer hold when you have used the outcomes to select predictors. There has been some work on this in recent years, introduced for example in Chapters 16 and 20 of Computer Age Statistical Inference. Under some assumptions it is possible to estimate p-values, but I think that it's safe to say this is still an area of active research interest. Unless you are willing to get into these issues in depth, it might be best to stay away from p-values for individual coefficients in LASSO.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ About the correlation of predictors: Meinshausen/Bühlmann claim that the subsetting of the sample helps with letting features "shine" even if there correlated with more prominent features for the whole sample. Also, they add a randomized scaling to features in every run because of possibly correlated features, which I didn't explain here. $\endgroup$ – Edgar Apr 17 at 14:55
  • $\begingroup$ @Edgar I agree that, insofar as it makes sense to evaluate importance among predictors in LASSO, the stability selection method is a promising approach. My reason for providing this answer is my fear that the OP or others who come upon this page might not have thought through what feature importance means in practice with multiple correlated predictors. $\endgroup$ – EdM Apr 17 at 15:02
  • $\begingroup$ Can I ask what is that parameter that you told that should be at least 0.6? is the lasso tuning parameter? or something I should calculate after the model is built? $\endgroup$ – Tiago Dias Apr 18 at 8:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.