12
$\begingroup$

I am running a regression model both with Lasso and Ridge (to predict a discrete outcome variable ranging from 0-5). Before running the model, I use SelectKBest method of scikit-learn to reduce the feature set from 250 to 25. Without an initial feature selection, both Lasso and Ridge yield to lower accuracy scores [which might be due to the small sample size, 600]. Also, note that some features are correlated.

After running the model, I observe that the prediction accuracy is almost the same with Lasso and Ridge. However, when I check first 10 features after ordering them by the absolute value of coefficients, I see that there is at most %50 overlap.

That is, given that different importance of features were assigned by each method, I might have a totally different interpretationbased on the model I choose.

Normally, the features represent some aspects of user behavior in a web site. Therefore, I want to explain the findings by highlighting the features (user behaviors) with stronger predictive ability vs weaker features (user behaviors). However, I do not know how to move forward at this point. How should I approach to interpreting the model? For example, should combine both and highlight the overlapping one, or should I go with Lasso since it provides more interpretability?

Improve this question
$\endgroup$
12
  • 3
    $\begingroup$ (+1) Regularization can be seen as making individual coefficient estimates worse while improving their collective performance at predicting new responses. What precisely are you trying to achieve with your interpretation? $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Mar 14, 2017 at 11:26
  • 1
    $\begingroup$ @Scortchi thanks for responding. I added this Normally, the features represent some aspects of user behavior in a web site. Therefore, I want to explain the findings by highlighting the features (user behaviors) with stronger predictive ability vs weaker features (user behaviors) . $\endgroup$
    – renakre
    Mar 14, 2017 at 11:40
  • 3
    $\begingroup$ +1 AFAIK the relation between ridge coefficients and lambda doesn't have to be monotonic, while in lasso it is. Thus, at certain shrinkage levels absolute value of coefficients in ridge and lasso may vary a lot. Having said that, I would appreciate if someone can sketch a proof of this or shortly explain it mathematically $\endgroup$
    – Łukasz Grad
    Mar 14, 2017 at 11:43
  • $\begingroup$ Make sure you are sorting the "beta" coefficients. See stats.stackexchange.com/a/243439/70282 You can get them by training on standardized variables or by adjustment later as described in the link. $\endgroup$
    – Chris
    Mar 14, 2017 at 12:57
  • 1
    $\begingroup$ @ŁukaszGrad LASSO coefficients need not be monotonic functions of $\lambda$ if predictors are correlated; see figure 6.6 of ISLR for an example. $\endgroup$
    – EdM
    Mar 18, 2017 at 15:18

1 Answer 1

Reset to default
7
$\begingroup$

Ridge regression encourages all coefficients to becomes small. Lasso encourages many/most[**] coefficients to become zero, and a few non-zero. Both of them will reduce the accuracy on the training set, but improve prediction in some way:

  • ridge regression attempts to improve generalization to the testing set, by reducing overfit
  • lasso will reduce the number of non-zero coefficients, even if this penalizes performance on both training and test sets

You can get different choices of coefficients if your data is highly correlated. So, you might have 5 features that are correlated:

  • by assigning small but non-zero coefficients to all of these features, ridge regression can achieve low loss on training set, which might plausibly generalize to testing set
  • lasso might choose only one single one of these, that correlates well with the other four. and there's no reason why it should pick the feature with highest coefficient in the ridge regression version

[*] for a definition of 'choose' meaning: assigns a non-zero coefficient, which is still a bit hand-waving, since ridge regression coefficients will tend to all be non-zero, but eg some might be like 1e-8, and others might be eg 0.01

[**] nuance: as Richard Hardy points out, for some use-cases, a value of $\lambda$ can be chosen which will result in all LASSO coefficients being non-zero, but with some shrinkage

Improve this answer
$\endgroup$
11
  • $\begingroup$ Good suggestions. A good check out be to do a correlation matrix. The non-overlapping variables may be highly correlated. $\endgroup$
    – Chris
    Mar 14, 2017 at 12:59
  • 3
    $\begingroup$ Good answer! However, I'm not sure it's fair to suggest that ridge universally attempts to impove test performance while not saying the same for lasso. For instance, if the true model is sparse (and in the subset of our predictors), we can immediately expect lasso to have better test performance than ridge $\endgroup$
    – user795305
    Mar 14, 2017 at 22:47
  • $\begingroup$ This is the 'bet on sparsity' principle. For instance, see the first plot here: faculty.bscb.cornell.edu/~bien/simulator_vignettes/lasso.html $\endgroup$
    – user795305
    Mar 14, 2017 at 22:51
  • 2
    $\begingroup$ Comparisons of variable choices (LASSO) and regression coefficients among multiple bootstrap samples of the data can nicely illustrate these issues. With correlated predictors, those chosen by LASSO from different bootstraps can be quite different while still providing similar predictive performance. Ideally, the entire model-building process including the initial feature-set reduction should be repeated on multiple bootstraps to document the quality of the process. $\endgroup$
    – EdM
    Mar 18, 2017 at 15:15
  • $\begingroup$ by choosing 4 of these features, with lowish coefficients, or even all of them, again with small, but non-zero, coefficients, ridge regression can low loss on training set -- ridge regression does not choose variables. Also, for low values of $\lambda$, lasso will choose all variables but do some shrinkage, just like ridge. $\endgroup$
    – Richard Hardy
    Mar 24, 2017 at 12:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.