4
$\begingroup$

Logistic regression with lasso versus PCA?

Got asked this question in an interview. I know the main difference is that Lasso is a regularization technique (adding vars to minimize effect of large coefficients) while PCA is feature selection technique (by covariance matrix decomposition).

I answered PCA allows you to do feature selection outside of the fit and transform and therefore give more flexibility in the hyper parameter search. Whereas in lasso the "feature selection" is kind of done for you and therefore there is less scope of hyper parameter optimization.

Does that sound right?

Improve this question
$\endgroup$
  • $\begingroup$ What exactly was the question? To compare and contrast PCA and LASSO? $\endgroup$ – Matthew Drury Oct 11 '16 at 3:16
  • $\begingroup$ I said I would use PCA for dimensionality reduction. The interviewer directly asked what is the difference between PCA and logistic reg. with lasso? $\endgroup$ – Manas Oct 11 '16 at 3:55
  • $\begingroup$ PCA isn't really feature selection because you lose the interpretability of your original features after reducing dimensions. PCA is a projection onto a lower dimensional linear subspace. That's it. What you "do" with that (say select the top k < n dimensions) is up to you. $\endgroup$ – ilanman Oct 11 '16 at 12:58
4
$\begingroup$

I answered PCA allows you to do feature selection outside of the fit and transform and therefore give more flexibility in the hyper parameter search.

PCA can be used as a dimensionality reduction technique if you drop Principal Components based on a heuristic, but it offers no feature selection, as the Principal Components are retained instead of the original features. However, tuning the number of Principal Components retained should work better than using heuristics, unless there are many low variance components and you are simply interested in filtering them.

Whereas in lasso the "feature selection" is kind of done for you and therefore there is less scope of hyper parameter optimization.

LASSO ($\ell_1$ regularization) on the other hand can, intrinsically, perform feature selection as the coefficients of predictors are shrunk towards zero. It still requires hyperparameter tuning because there's a regularization coefficient that weights how severe is the regularization of the loss function.

As @MatthewDrury commented, ordinary PCA is agnostic to the target variable while LASSO regression isn't, as it's part of a regression model. This is the most important difference, actuallly.

Improve this answer
$\endgroup$
  • 3
    $\begingroup$ I think a complete answer really needs to mention that using PCA for dim reduction in regression ignores the relationship between $X$ and $y$. So dropping low variance components while ignoring their relationship to $y$ is a dubious idea. $\endgroup$ – Matthew Drury Oct 11 '16 at 15:16
  • 1
    $\begingroup$ @MatthewDrury Thanks for the comment, yeah, that's the most important point actually, will expand on that. $\endgroup$ – Firebug Oct 11 '16 at 15:18
  • 1
    $\begingroup$ +1 but if I were an interviewer I would like to hear about the relationships of "PC regression" to "ridge regression", and then about the lasso vs ridge/PCA. It's true that PCA on $X$ ignores $y$ as @MatthewDrury said, but then ridge regression does not ignore it but one can show that PC regression and RR are closely related. Both penalize low variance directions in X. $\endgroup$ – amoeba Oct 11 '16 at 16:06
  • 1
    $\begingroup$ @amoeba If you could do all that in an interview you'd be well on your way to being hired by me. $\endgroup$ – Matthew Drury Oct 11 '16 at 17:58
  • $\begingroup$ @MatthewDrury I get the point that PCA ignores relationship between X and y where as Lasso (l1 regularization) does not. Thanks! $\endgroup$ – Manas Oct 11 '16 at 20:31
0
$\begingroup$

  • PCA while reducing the number of features does not care about the class labels. The only thing that it cares about is preserving the maximum variance which may not always be optimal for classification task.

  • L1-Reg on the other hand pushes those features towards zero that do not have much correlation with the class labels. Hence, L1-Reg strives to reduce the number of features while also getting good classification performance.

  • To avoid under-fitting, we can always do hyper-parameter tuning to find best lambda.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.