5
$\begingroup$

I'm trying to estimate how much feature reduction using PCA can help with increasing accuracy in case of classification using different ml methods. I'm using digits dataset available in scikit-learn. To do it, I'm checking accuracy using 64 features available, later using PCA, I reduce it to 63 features and accuracy decreases extremely:

###ANN:

featureNum #accuracy

64  | 0.966 +- 0.008

63  | 0.132 +- 0.0116619037897

###SVM:

featureNum accuracy

64  | 0.96 +-   0.0

63  | 0.54 +-   0.0

###RandomForest:

featureNum accuracy

64  | 0.974 +- 0.008

63  | 0.12 +-   0.022803508502

###DecisiontTree:

featureNum accuracy

64 |    0.802 +- 0.0172046505341

63 |    0.11 +- 0.0126491106407

All calculations were repeated 5 times to get statistics. Before using PCA (64 features) scores where quite good in all cases. After, In case of all tested methods apart from SVM, it was practically random (there're 10 classes). I would understand that accuraccy dropped a little because we loose some information for sure using PCA but it's quite extreme. 64 features is quite a lot so I rather expected that accuracy can increase. I tried also with dataset create with make_classifiation using 100 features and but it didn't change much. Described results I got when using 1000 records from digits dataset, I tried with lower amount of data but still it's the same results more or less.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Please show the code. It's most likely there is an error in your code - I was able to get exactly the same accuracy of SVMs using such PCA. $\endgroup$
    – Jakub Bartczuk
    Commented Sep 17, 2017 at 20:02
  • $\begingroup$ Are you sure you are not doing fit_transform for both training and test dataset and use different classifiers for 64 and 63-dimensional data? $\endgroup$
    – Jakub Bartczuk
    Commented Sep 17, 2017 at 20:03
  • $\begingroup$ Another possibility: stats.stackexchange.com/questions/52773/… $\endgroup$
    – Cam.Davidson.Pilon
    Commented Sep 23, 2017 at 2:39

2 Answers 2

Reset to default
14
$\begingroup$

I've created a notebook that almost replicates your drop in accuracy.

I think that most likely error is actually retraining PCA - if you fit PCA on train set, then fit classifier, and then try to run it on principal components retrieved from the test set, then you use incorrect parameter space for the classifier - classifier uses train set principal components as coordinates, and then you run it on test set PCs.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ That's it! In my code I had method def reduce_dimensions(x_train, y_train, x_test, y_test, reduction_object)that was performing PCA seperately on x_test and x_train I changed it so as not to refit PCA and now it makes sense. You're magican that you could see it without looking into the code Thanks a lot Jakub! : ) $\endgroup$
    – PawelPawel
    Commented Sep 17, 2017 at 21:13
3
$\begingroup$

I think you have a wrong hypothesis to verify.

In general, applying PCA before building a model will NOT help to make the model perform better (in terms of accuracy)!

This is because PCA is an algorithm that does not consider the response variable / prediction target into account. PCA will treat the feature has large variance as important features, but the feature has large variance can have noting to do with the prediction target.

This means, you can produce a lot of useless features and eliminate useful features after PCA.

Please check my answer here for details and some demo.

How to decide between PCA and logistic regression?

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you for advice. I guess, I'll take it into consideration anyway and I'll compare it with LDA which takes response into account and it'll be good example to describe it in my thesis : ) $\endgroup$
    – PawelPawel
    Commented Sep 18, 2017 at 6:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.