2
$\begingroup$

When dealing with the curse of dimensionality, regularization methods seem to be clear in their intuition. All "regularization" methods can be seen as a "squeezing" of one's variables towards 0, when there exists too many of them.

However, PCA dimensionality reduction is another way to handle the same curse of dimensionality, but it does not seem that PCA possesses any comparable interpretation.

Essentially, does a similar intuitive interpretation exist for PCA as for regularization, which makes the former comparable to the latter?

Any insights placing both frameworks on the same meta-footing would be appreciated.

$\endgroup$
  • 1
    $\begingroup$ PCA regularization is very similar to the ridge regularization. See here: stats.stackexchange.com/a/133257/28666 $\endgroup$ – amoeba Jun 25 at 6:35
  • $\begingroup$ I've gone through that answer before, and, quite frankly, it is a bit complex to grasp through all the math (which is nice, but not helpful in my context). $\endgroup$ – Coolio2654 Jun 25 at 6:46
5
$\begingroup$

PCA considers only the variance of the features ($X$) but not the relationship between features and labels while doing this compression. Regularization, on the other hand, acts directly on the relationship between features and labels and hence develops models which are better at explaining the labels given the features.

I'm not familiar with other fields but Finance literature, in particular Shrinking the Cross Section, has done this comparison (PCA of features vs regularized model) and found that regularized model does a better job of predicting portfolio returns (better out-of-sample $R^2$)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.