5
$\begingroup$

According to Wikipedia page on Naive Bayes:

.. Naive Bayes classifiers are a family of simple "probabilistic classifiers" based on applying Bayes' theorem with strong (naive) independence assumptions between the features.

Since data features may not be independent of each other, should one always perform PCA before applying Naive Bayes? PCA is expected to create components which are not much correlated with each other and hence one can expect more robust results with Naive Bayes.

$\endgroup$
3
  • $\begingroup$ "PCA is expected to create components which are not much correlated with each other.." - do principal components are supposed to have any correlation? $\endgroup$
    – hafiz031
    Commented Mar 1, 2022 at 4:02
  • $\begingroup$ You should ask this as a separate question. Also post the link of that question here. $\endgroup$
    – rnso
    Commented Mar 1, 2022 at 7:13
  • $\begingroup$ @mso probably I have found what I was looking for: stats.stackexchange.com/q/153928/245577 and stats.stackexchange.com/q/110508/245577 show PCs should be uncorrelated. $\endgroup$
    – hafiz031
    Commented Mar 3, 2022 at 7:18

1 Answer 1

5
$\begingroup$

For general cases, I don't think doing PCA first will improve the classification results for the Naive Bayes classifier. Naive Bayes assumes the features are conditional independent, which means given the class, $p(x_i|C_k)=p(x_{i}|x_{i+1}...x_n,C_k)$, this does not mean that the features have to be independent.

Moreover, I don't think PCA can improve the conditional independence in general. Using PCA without dimension reduction is just doing coordinate rotation, without taken into account the discrimination power between different class. And in most of the cases this rotation won't give uncorrelated features for each class, as shown in this following figure. enter image description here And using PCA to do dimension reduction, this might even worse the situation when the feature with discrimination power has small variance and is threw away by doing PCA first.

$\endgroup$
1
  • $\begingroup$ Your last objection re - "feature with small variance being thrown away" - can possibly be overcome by scaling before PCA? $\endgroup$
    – rnso
    Commented Sep 23, 2020 at 16:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.