# Logistic regression with lasso versus PCA?

Logistic regression with lasso versus PCA?

Got asked this question in an interview. I know the main difference is that Lasso is a regularization technique (adding vars to minimize effect of large coefficients) while PCA is feature selection technique (by covariance matrix decomposition).

I answered PCA allows you to do feature selection outside of the fit and transform and therefore give more flexibility in the hyper parameter search. Whereas in lasso the "feature selection" is kind of done for you and therefore there is less scope of hyper parameter optimization.

Does that sound right?

• What exactly was the question? To compare and contrast PCA and LASSO? Oct 11, 2016 at 3:16
• I said I would use PCA for dimensionality reduction. The interviewer directly asked what is the difference between PCA and logistic reg. with lasso? Oct 11, 2016 at 3:55
• PCA isn't really feature selection because you lose the interpretability of your original features after reducing dimensions. PCA is a projection onto a lower dimensional linear subspace. That's it. What you "do" with that (say select the top k < n dimensions) is up to you. Oct 11, 2016 at 12:58

I answered PCA allows you to do feature selection outside of the fit and transform and therefore give more flexibility in the hyper parameter search.

PCA can be used as a dimensionality reduction technique if you drop Principal Components based on a heuristic, but it offers no feature selection, as the Principal Components are retained instead of the original features. However, tuning the number of Principal Components retained should work better than using heuristics, unless there are many low variance components and you are simply interested in filtering them.

Whereas in lasso the "feature selection" is kind of done for you and therefore there is less scope of hyper parameter optimization.

LASSO ($\ell_1$ regularization) on the other hand can, intrinsically, perform feature selection as the coefficients of predictors are shrunk towards zero. It still requires hyperparameter tuning because there's a regularization coefficient that weights how severe is the regularization of the loss function.

As @MatthewDrury commented, ordinary PCA is agnostic to the target variable while LASSO regression isn't, as it's part of a regression model. This is the most important difference, actuallly.

• I think a complete answer really needs to mention that using PCA for dim reduction in regression ignores the relationship between $X$ and $y$. So dropping low variance components while ignoring their relationship to $y$ is a dubious idea. Oct 11, 2016 at 15:16
• @MatthewDrury Thanks for the comment, yeah, that's the most important point actually, will expand on that. Oct 11, 2016 at 15:18
• +1 but if I were an interviewer I would like to hear about the relationships of "PC regression" to "ridge regression", and then about the lasso vs ridge/PCA. It's true that PCA on $X$ ignores $y$ as @MatthewDrury said, but then ridge regression does not ignore it but one can show that PC regression and RR are closely related. Both penalize low variance directions in X. Oct 11, 2016 at 16:06
• @amoeba If you could do all that in an interview you'd be well on your way to being hired by me. Oct 11, 2016 at 17:58
• @davidhigh With ridge you still need the original X. With PCA you can discard it altogether. That's the difference. Nov 21, 2018 at 7:04
• PCA while reducing the number of features does not care about the class labels. The only thing that it cares about is preserving the maximum variance which may not always be optimal for classification task.

• L1-Reg on the other hand pushes those features towards zero that do not have much correlation with the class labels. Hence, L1-Reg strives to reduce the number of features while also getting good classification performance.

• To avoid under-fitting, we can always do hyper-parameter tuning to find best lambda.