# Why does Naive Bayes work better when the number of features >> sample size compared to more sophisticated ML algorithms?

Because of the class independence assumption, naive Bayes classifiers can quickly learn to use high dimensional features with limited training data compared to more sophisticated methods. This can be useful in situations where the dataset is small compared to the number of features, such as images or texts.

Why does Naive Bayes work well when the number of features >> sample size compared to more sophisticated ML algorithms?

What the author is getting at is that Naive Bayes implicitly treats all features as being independent of one another, and therefore the sorts of curse-of-dimensionality problems which typically rear their head when dealing with high-dimensional data do not apply.

If your data has $$k$$ dimensions, then a fully general ML algorithm which attempts to learn all possible correlations between these features has to deal with $$2^k$$ possible feature interactions, and therefore needs on the order of $$2^k$$ many data points to be performant. However because Naive Bayes assumes independence between features, it only needs on the order of $$k$$ many data points, exponentially fewer.

However this comes at the cost of only being able to capture much simpler mappings between the input variables and the output class, and as such Naive Bayes could never compete with something like a large neural network trained on a large dataset when it comes to tasks like image recognition, although it might perform better on very small datasets.

• Thanks. Can you link some paper or source which back this reasoning? Nov 29 '18 at 13:22
• For someone whose handle is ML_Pro, this is something you should probably already know ;) inf.ed.ac.uk/teaching/courses/inf2b/learnnotes/… Nov 29 '18 at 15:43
• I guess you are happy now :D Just curious, can you guess what my handle mean? Nov 30 '18 at 10:46
• That's a great handle! George of the Jungle (random forest) pun! Nov 30 '18 at 10:56
• Yes. That is what I meant! Nov 30 '18 at 10:58