According to this article

Because of the class independence assumption, naive Bayes classifiers can quickly learn to use high dimensional features with limited training data compared to more sophisticated methods. This can be useful in situations where the dataset is small compared to the number of features, such as images or texts.

Why does Naive Bayes work well when the number of features >> sample size compared to more sophisticated ML algorithms?


1 Answer 1


What the author is getting at is that Naive Bayes implicitly treats all features as being independent of one another, and therefore the sorts of curse-of-dimensionality problems which typically rear their head when dealing with high-dimensional data do not apply.

If your data has $k$ dimensions, then a fully general ML algorithm which attempts to learn all possible correlations between these features has to deal with $2^k$ possible feature interactions, and therefore needs on the order of $2^k$ many data points to be performant. However because Naive Bayes assumes independence between features, it only needs on the order of $k$ many data points, exponentially fewer.

However this comes at the cost of only being able to capture much simpler mappings between the input variables and the output class, and as such Naive Bayes could never compete with something like a large neural network trained on a large dataset when it comes to tasks like image recognition, although it might perform better on very small datasets.

  • $\begingroup$ Thanks. Can you link some paper or source which back this reasoning? $\endgroup$ Nov 29, 2018 at 13:22
  • $\begingroup$ For someone whose handle is ML_Pro, this is something you should probably already know ;) inf.ed.ac.uk/teaching/courses/inf2b/learnnotes/… $\endgroup$
    – jon_simon
    Nov 29, 2018 at 15:43
  • 1
    $\begingroup$ I guess you are happy now :D Just curious, can you guess what my handle mean? $\endgroup$ Nov 30, 2018 at 10:46
  • 1
    $\begingroup$ That's a great handle! George of the Jungle (random forest) pun! $\endgroup$
    – jon_simon
    Nov 30, 2018 at 10:56
  • $\begingroup$ Yes. That is what I meant! $\endgroup$ Nov 30, 2018 at 10:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.