8
$\begingroup$

I have seen PCA being used to eliminate variables for dimensionality reduction, but LASSO's ability to eliminate variables for the very same purpose is often described as a disadvantage.

My intuition is that in PCA we still retain some of the information in the reduced dimensions as the newly formed features are linear combinations of the old variables, whereas in LASSO we just completely drop some of the original variables by setting the coefficients of non selected features to zero. Is my intuition correct? If not, then why is dimensionality reduction a drawback in one scenario and an advantage in the other?

Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
6
$\begingroup$

Principal Components Analysis (PCA) doesn't select among the original features unless the features are orthogonal. It lowers the dimension of the feature space by ignoring PCs (linear combinations of potentially all the predictors) that explain little of the feature-set variance, but in general it does not remove any of the original features from the problem.

Some find the ability of LASSO to choose among several correlated features to be a strength rather than a weakness, particularly when the number of features greatly exceeds the number of cases. Yes, LASSO's choice among correlated features can be highly sample dependent. Nevertheless, as LASSO penalizes the coefficients for the retained variables it can still be useful for developing prediction models. The "disadvantage" is when one tries to interpret the selection by LASSO in terms of "variable importance" for inference.

The issues raised in this question point out a potential advantage of ridge regression, which is effectively PCA with varying weights among the PCs rather than simple 0/1 weights. (See page 79 of ESLII.) Try both LASSO and ridge regression on multiple bootstrap samples of a data set having correlated predictors. The particular features selected by LASSO may change dramatically among the bootstrap samples. This illustrates the mistake of interpreting choices by LASSO in terms of an underlying true "variable importance." The coefficients of correlated predictors in ridge regression may be variable among bootstrap samples but the ridge models will contain all the features.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I'm a bit confused about your argument in favor of Ridge regression against LASSO in the last paragraph. While I agree that one shouldn't interpret LASSO as assessing each variable's importance for highly correlated predictors, are you saying that among different bootstrap samples, the coefficient variability in Ridge is never greater than LASSO? I would guess that depending on the size of the regularization $\lambda$, the primary difference would be that LASSO may change which features are set to 0, while Ridge changes which features are assigned coefficients close to 0 - so what? $\endgroup$
    – Don Walpola
    Sep 3, 2018 at 0:50
  • $\begingroup$ @DonWalpola I see my last paragraph more as a warning than as an argument against LASSO. From a set of correlated predictors, any predictor-selection method will choose the one that happened to work best on the data sample at hand. The selected predictor might not work so well on a new data set, while one of those omitted might have worked better on the new data. Unlike LASSO, ridge doesn't throw away information about correlated predictors and might mitigate somewhat against this phenomenon. So with a moderate number of predictors at least, why choose LASSO over ridge for prediction? $\endgroup$
    – EdM
    Sep 3, 2018 at 16:58
  • $\begingroup$ Well, I guess my question has to do with how Ridge performs regularization, and if you can really say that it doesn't throw out information. I don't have much experience with using Ridge, but my understanding is that there would still be a significant difference in the magnitude of the coefficients assigned to each predictor in a correlated set. If my understanding is correct, wouldn't that mean that the same risk of misinterpreting 'importance' based on coefficient magnitude is present in Ridge, even though you technically haven't 'thrown away' any variables? $\endgroup$
    – Don Walpola
    Sep 3, 2018 at 17:05
  • $\begingroup$ @DonWalpola you are correct that there will be sample-dependent differences in magnitudes of coefficients within a set of correlated predictors, but at least all are included into the final model. In the context of prediction, whether LASSO or ridge will perform better may depend on the scale of the data and the numbers of predictors available; the hybrid elastic net can combine their strengths. In the context of inference, my warning would be always to avoid assigning 'importance' to predictors based either on what LASSO selected or on the relative magnitudes of ridge coefficients. $\endgroup$
    – EdM
    Sep 3, 2018 at 18:33
  • $\begingroup$ Alright, thanks for the clarification. I think I was just confused because I didn't see you explicitly make reference to possible inferential pitfalls that occur in regularization techniques used for 'variable selection/importance' in general in your original post. I wonder, once scale normalization is performed, would the rotational invariance of the $L_{2}$ norm provide a means to demonstrate whether or not the coefficients from ridge regression suffered from a similar magnitude disparity as those from LASSO? $\endgroup$
    – Don Walpola
    Sep 3, 2018 at 18:43
0
$\begingroup$

Using PCA to do feature selection is different than using LASSO because PCA does not consider the response variable, but only check the variance in features.

See my answers here, with graphical illustration (logistic regression case / linear method without regularization).

How to decide between PCA and logistic regression?

Improve this answer
$\endgroup$
1
  • $\begingroup$ I know that, but my question is of different taste. I want to know why is dropping some correlated variables is advantage in one and disadvantage in the other irrespective of whether they use the response or not. $\endgroup$
    – Siddharth Shakya
    May 21, 2017 at 17:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.