I am performing a logistic regression on a rather big dataset (700k+ samples and 1k+ features). I suspect that a lot of these features will be highly correlated and multicollinearity can be an issue.

I believe that with elastic net regularisation is can perform feature selection and shrink the parameters of correlated features toghether (to prevent overfitting)

I understand that after PCA I end up with new (linnearly) uncorrelated (orthonal) features. I'm I correct to think that I now only need an L1 regularisation term in my optimazation function to have a simmular model as in the case with elastic net?

I prefer to do the PCA because I can reduce the number of parameters from 1k+ to around 60. (the rest of the PCs have 0 variance) which is more manageble to work with.

I am only interesed in a developing a prediction algorithm. I use R.

Best regards

  • $\begingroup$ Feature reduction by PCA is not similar to L2 regularization. Be careful by blindly applying PCA. It is e.g. extremely sensitive to outliers or heavily skewed distributions. $\endgroup$ – Michael M Aug 3 '14 at 17:05
  • $\begingroup$ If I suddenly jump from 10^6 to 10^13 % explaining variance as indicated by the eigenvalues ath the 63th PC then isn't it save to say that all information is in the first 63 PC and the rest can be disgarded in subsequent analysis? Also, another reason to get rid of L2 regularisation is that its a parameter less to fiddle with. $\endgroup$ – statastic Aug 4 '14 at 5:47
  • $\begingroup$ A single outlier can make the first pc explain 99% of the variance. $\endgroup$ – Michael M Aug 4 '14 at 5:51
  • $\begingroup$ Upon visual inspections (histograms of the loadings) this seems to not be the case for the majority. There are some PC that seem to be influenced by extreme values but I hope these will get removed by the L1 regularization. $\endgroup$ – statastic Aug 4 '14 at 7:01
  • $\begingroup$ I don't think they are equivalent. Why don't you still use both L2 and L1 after PCA? $\endgroup$ – amoeba Dec 17 '14 at 1:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.