According to this article

Because of the class independence assumption, naive Bayes classifiers can quickly learn to use high dimensional features with limited training data compared to more sophisticated methods. This can be useful in situations where the dataset is small compared to the number of features, such as images or texts.

Why does Naive Bayes work well when the number of features >> sample size compared to more sophisticated ML algorithms?


What the author is getting at is that Naive Bayes implicitly treats all features as being independent of one another, and therefore the sorts of curse-of-dimensionality problems which typically rear their head when dealing with high-dimensional data do not apply.

If your data has $k$ dimensions, then a fully general ML algorithm which attempts to learn all possible correlations between these features has to deal with $2^k$ possible feature interactions, and therefore needs on the order of $2^k$ many data points to be performant. However because Naive Bayes assumes independence between features, it only needs on the order of $k$ many data points, exponentially fewer.

However this comes at the cost of only being able to capture much simpler mappings between the input variables and the output class, and as such Naive Bayes could never compete with something like a large neural network trained on a large dataset when it comes to tasks like image recognition, although it might perform better on very small datasets.

  • $\begingroup$ Thanks. Can you link some paper or source which back this reasoning? $\endgroup$ – GeorgeOfTheRF Nov 29 '18 at 13:22
  • $\begingroup$ For someone whose handle is ML_Pro, this is something you should probably already know ;) inf.ed.ac.uk/teaching/courses/inf2b/learnnotes/… $\endgroup$ – jon_simon Nov 29 '18 at 15:43
  • 1
    $\begingroup$ I guess you are happy now :D Just curious, can you guess what my handle mean? $\endgroup$ – GeorgeOfTheRF Nov 30 '18 at 10:46
  • 1
    $\begingroup$ That's a great handle! George of the Jungle (random forest) pun! $\endgroup$ – jon_simon Nov 30 '18 at 10:56
  • $\begingroup$ Yes. That is what I meant! $\endgroup$ – GeorgeOfTheRF Nov 30 '18 at 10:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.