3
$\begingroup$

I have a dataset which originally had 34000 features but only 4968 samples of data. To which I applied PCA and derived 4968 principal components.

Here is a table detailing the principal components derived and their respective cumulative explained variance

Table of explained variance

So my question is this. Why do models start to perform extremely poor when i choose K components which amount close to or equal to 100% explained variance?

Some example results I have logged

  • Linear SVM

Linear SVM PCA

  • Logistic Regression

enter image description here

$\endgroup$
5
$\begingroup$

Presumably there is a degree of noise in your original 34k features dataset?

In which case the full rank PCA is modelling that noise in the smaller (higher numbered) components, and this results in and over-fitting of the classifier.

A typical way to account for this would be to use cross-validation of the PCA models to generate a Q2 score that would tell you at what point your components were fitting to noise and not use these in your model.

$\endgroup$
  • $\begingroup$ I had a separate set of test data which are unseen data compared to the training data. So after either SVMs were trained with training data, CV at 10-fold was performed using the trained model at that point. Can the CV accuracy be relied on to tell me that it is actually over-fitting at the higher PC levels? (Linear SVM validation accuracy is from conducting CV) $\endgroup$ – Marcus Lim Jul 16 '18 at 19:55
4
$\begingroup$

With about 5000 observations and assuming $Y$ is binary with 2000 events, one rule of thumb is that you can model about 2000/15 = 133 parameters. So going beyond 133 PCs presents risk of overfitting. In general I would not go anywhere near the 133 but would try the first 20 PCs. And note that PCs after the first few can be quite unstable.

This doesn't help with stability but does help with interpretation: sparse PCs.

You are using improper accuracy scoring rules, which will mislead you on every front. See here and here.

$\endgroup$
  • $\begingroup$ ... and in any case, OP could monitor stability of the models as function of #PCs. $\endgroup$ – cbeleites Jul 15 '18 at 18:25
  • $\begingroup$ @frank could you indicate on which accuracy do you refer to that i have used improper accuracy scoring? For CV accuracy, i used scikitlearn's cross_val_score to derive this. For test accuracy, it was conducted on unseen data and derived from the model's score function. I manually calculated the score from log probability prediction over truth labels and got the same accuracy score above. Am i misunderstanding what you meant by improper scoring rules? $\endgroup$ – Marcus Lim Jul 17 '18 at 9:52
  • $\begingroup$ @frank i also do not understand how you derived 15 to divide by for getting 133 parameters. $\endgroup$ – Marcus Lim Jul 17 '18 at 10:03
  • $\begingroup$ Proportion classified correctly and all other simple proportions are improper accuracy scores. You didn't look at the links I provided. The 15:1 rule is a very rough rule of thumb for avoiding overfitting as described in my Regression Modeling Strategies book and course notes. There are many better rules being developed, and you need a certain sample size just to estimate the intercept, as I discuss in the book. $\endgroup$ – Frank Harrell Jul 17 '18 at 11:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.