3
$\begingroup$

I have a dataset which originally had 34000 features but only 4968 samples of data. To which I applied PCA and derived 4968 principal components.

Here is a table detailing the principal components derived and their respective cumulative explained variance

Table of explained variance

So my question is this. Why do models start to perform extremely poor when i choose K components which amount close to or equal to 100% explained variance?

Some example results I have logged

  • Linear SVM

Linear SVM PCA

  • Logistic Regression

enter image description here

Improve this question
$\endgroup$

2 Answers 2

Reset to default
5
$\begingroup$

Presumably there is a degree of noise in your original 34k features dataset?

In which case the full rank PCA is modelling that noise in the smaller (higher numbered) components, and this results in and over-fitting of the classifier.

A typical way to account for this would be to use cross-validation of the PCA models to generate a Q2 score that would tell you at what point your components were fitting to noise and not use these in your model.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I had a separate set of test data which are unseen data compared to the training data. So after either SVMs were trained with training data, CV at 10-fold was performed using the trained model at that point. Can the CV accuracy be relied on to tell me that it is actually over-fitting at the higher PC levels? (Linear SVM validation accuracy is from conducting CV) $\endgroup$
    – Marcus Lim
    Jul 16, 2018 at 19:55
4
$\begingroup$

With about 5000 observations and assuming $Y$ is binary with 2000 events, one rule of thumb is that you can model about 2000/15 = 133 parameters. So going beyond 133 PCs presents risk of overfitting. In general I would not go anywhere near the 133 but would try the first 20 PCs. And note that PCs after the first few can be quite unstable.

This doesn't help with stability but does help with interpretation: sparse PCs.

You are using improper accuracy scoring rules, which will mislead you on every front. See here and here.

Improve this answer
$\endgroup$
4
  • $\begingroup$ ... and in any case, OP could monitor stability of the models as function of #PCs. $\endgroup$
    – cbeleites unhappy with SX
    Jul 15, 2018 at 18:25
  • $\begingroup$ @frank could you indicate on which accuracy do you refer to that i have used improper accuracy scoring? For CV accuracy, i used scikitlearn's cross_val_score to derive this. For test accuracy, it was conducted on unseen data and derived from the model's score function. I manually calculated the score from log probability prediction over truth labels and got the same accuracy score above. Am i misunderstanding what you meant by improper scoring rules? $\endgroup$
    – Marcus Lim
    Jul 17, 2018 at 9:52
  • $\begingroup$ @frank i also do not understand how you derived 15 to divide by for getting 133 parameters. $\endgroup$
    – Marcus Lim
    Jul 17, 2018 at 10:03
  • $\begingroup$ Proportion classified correctly and all other simple proportions are improper accuracy scores. You didn't look at the links I provided. The 15:1 rule is a very rough rule of thumb for avoiding overfitting as described in my Regression Modeling Strategies book and course notes. There are many better rules being developed, and you need a certain sample size just to estimate the intercept, as I discuss in the book. $\endgroup$
    – Frank Harrell
    Jul 17, 2018 at 11:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.