19
$\begingroup$

I already have an idea about pros and cons of ridge regression and the LASSO.

For the LASSO, L1 penalty term will yield a sparse coefficient vector, which can be viewed as a feature selection method. However, there are some limitations for the LASSO. If the features have high correlation, the LASSO will only select one of them. In addition, for problems where $p$ > $n$, the LASSO will select at most $n$ parameters ($n$ and $p$ are the number of observations and parameters, respectively). These make the LASSO empirically a suboptimal method in terms of predictability compared to ridge regression.

For ridge regression, it offers better predictability in general. However, its interpretability is not as nice as the LASSO.

The above explanation can often be found in textbooks in machine learning/data mining. However, I am still confused about two things:

  1. If we normalize the feature range (say between 0 and 1, or with zero mean and unit variance), and run ridge regression, we can still have an idea of feature importance by sorting the absolute values of coefficients (the most important feature has the highest absolute value of coefficients). Though we are not selecting features explicitly, interpretability isn't lost using ridge regression. At the same time, we can still achieve high prediction power. Then why do we need the LASSO? Am I missing something here?

  2. Is the LASSO preferred due to its feature selection nature? To my understanding, the reasons why we need feature selection are the ability to generalize and ease of computation.

    For ease of computation, we don't want to feed all 1 million features into our model if we are performing some NLP tasks, so we drop some obviously useless features first to reduce the computational cost. However, for the LASSO, we can only know the feature selection result (the sparse vector) after we feed all the data into our model, so we don't benefit from the LASSO in terms of reducing computational cost. We can only make prediction a little faster as now we only feed the subset of features (say 500 out of 1 million) into our model to generate predicted results.

    If the LASSO is preferred for its ability to generalize, then we can also achieve the same goal using ridge regression (or any other kind of regularization). Why do we need LASSO (or elastic nets) again? Why can't we just stick to ridge regression?

Could someone please shed some lights on this? Thanks!

Improve this question
$\endgroup$
5
  • 3
    $\begingroup$ These make the LASSO empirically a suboptimal method in terms of predictability compared to ridge regression. I disagree. I don't think LASSO is generally worse (or better) than ridge in terms of prediction. As @jona says in his/her answer, you might face situations where some of the feature truly don't belong to the model, and then LASSO will be more effective of kicking them out. However, with ridge all the features would be included and the irrelevant ones would contaminate predictions. That's why we need elastic net - to let the data decide the appropriate combination of $L_1$ and $L_2$. $\endgroup$
    – Richard Hardy
    Commented Nov 1, 2015 at 10:47
  • 3
    $\begingroup$ I also wonder which textbooks say things like For ridge regression, it offers better predictability in general (in contrast to LASSO, I understand, not in contrast to unrestricted regression). Perhaps general is not that general in their use. Also, how much interpretability are regularization methods supposed to yield? (Also, Shmueli "To Explain or To Predict" (2010) is a nice piece, even though not directly related.) $\endgroup$
    – Richard Hardy
    Commented Nov 1, 2015 at 10:51
  • 1
    $\begingroup$ @RichardHardy, you are right. Now I read the textbook more carefully and found that "neither ridge regression nor the lasso will universally dominate the other" on page 223, An Introduction to Statistical Learning with Applications in R, Gareth James et al $\endgroup$
    – Brad Li
    Commented Nov 1, 2015 at 16:35
  • $\begingroup$ @RichardHardy, originally I found similar arguments for L1-regularization on the LIBLINEAR FAQ: csie.ntu.edu.tw/~cjlin/liblinear/… $\endgroup$
    – Brad Li
    Commented Nov 1, 2015 at 16:40
  • $\begingroup$ Would runs of ridge and Lasso on a real example or two clarify the differences ? (But they're not easy to compare -- plot fit vs. sparsity ?) $\endgroup$
    – denis
    Commented Dec 13, 2015 at 12:07

2 Answers 2

Reset to default
21
$\begingroup$

  1. If you order 1 million ridge-shrunk, scaled, but non-zero features, you will have to make some kind of decision: you will look at the n best predictors, but what is n? The LASSO solves this problem in a principled, objective way, because for every step on the path (and often, you'd settle on one point via e.g. cross validation), there are only m coefficients which are non-zero.

  2. Very often, you will train models on some data and then later apply it to some data not yet collected. For example, you could fit your model on 50.000.000 emails and then use that model on every new email. True, you will fit it on the full feature set for the first 50.000.000 mails, but for every following email, you will deal with a much sparser and faster, and much more memory efficient, model. You also won't even need to collect the information for the dropped features, which may be hugely helpful if the features are expensive to extract, e.g. via genotyping.

Another perspective on the L1/L2 problem exposed by e.g. Andrew Gelman is that you often have some intuition what your problem may be like. In some circumstances, it is possible that reality is truly sparse. Maybe you have measured millions of genes, but it is plausible that only 30.000 of them actually determine dopamine metabolism. In such a situation, L1 arguably fits the problem better.
In other cases, reality may be dense. For example, in psychology, "everything correlates (to some degree) with everything" (Paul Meehl). Preferences for apples vs. oranges probably does correlate with political leanings somehow - and even with IQ. Regularization might still make sense here, but true zero effects should be rare, so L2 might be more appropriate.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Thanks. Your explanation is very clear! I am still a bit confused about ridge's interpretability. Can I define the feature importance by sorting the variables based on their absolute values? For example, if we obtain the following result using ridge $$y = -2x_{1} + 3x_{2}-x_{3}$$ we can say $x_{2} > x_{1} > x_{3}$ in terms of feature importance, given that we already normalize features within the range $[0, 1]$. Therefore we can still achieve interpretability using ridge. $\endgroup$
    – Brad Li
    Commented Nov 1, 2015 at 3:36
  • $\begingroup$ Of course you can sort them, but you will still have to make some sort of decision as to which subset of them you look at. $\endgroup$
    – jona
    Commented Nov 1, 2015 at 3:38
  • 6
    $\begingroup$ Another way to phrase this would be: ridge may help with feature selection, LASSO does feature selection. $\endgroup$
    – jona
    Commented Nov 1, 2015 at 13:20
  • 1
    $\begingroup$ @Brad, in addition to the excellent answer by jona (+1), note that judging feature importance by its standardized regression coefficient is one possible approach, but not the only one; there are different measures of "feature importance" and they can easily give contradictory results. See this thread for a long discussion: stats.stackexchange.com/questions/64010. $\endgroup$
    – amoeba
    Commented Nov 1, 2015 at 14:47
1
$\begingroup$

Interpretability decreases if the target is dependent on lot of features. It increases if we can reduce the number of features as well maintain the accuracy. Ridge regularization does not have the ability to reduce number of features. But Lasso has the ability. How this happens is explained visually in the following link:

Click Article on Towards Data Science

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.