In the naive bayes page in Scikit-Lean, one of differences between Bernoulli Naive Bayes and Multinomial Naive Bayes is that

it (bernoulli) explicitly penalizes the non-occurrence of a feature i that is an indicator for class y, where the multinomial variant would simply ignore a non-occurring feature.

I wanna ask the the reason of this difference.
IMHO, Bernoulli Naive Bayes should also ignore the non-occurrence features like what Multinomial Naive Bayes does.


I found the reason of this difference. The answer is the difference of feature distribution between these two models:

  1. In Bernoulli Naive Bayes, the features are viewed as independent binary variables. According to Naive Bayes' wikipedia,

    If $x_{i}$ is a boolean expressing the occurrence or absence of the i'th term from the vocabulary, then the likelihood of a document given a class $ C_{k}$ is given by $p(\mathbf {x} \mid C_{k})=\prod _{i=1}^{n}p_{ki}^{x_{i}}(1-p_{ki})^{(1-x_{i})}$ where $p_{{ki}}$ is the probability of class $ C_{k}$ generating the term $w_{i}$

    Therefore, every feature in the feature set should be calculated even when the feature is non-occurring.

  2. In Multinomial Naive Bayes, the features obeys a multinomial distribution. According to the wikipedia,

    A feature vector ${\mathbf {x}}=(x_{1},\dots ,x_{n})$ is then a histogram, with $x_{i}$ counting the number of times event i was observed in a particular instance. This is the event model typically used for document classification, with events representing the occurrence of a word in a single document (see bag of words assumption). The likelihood of observing a histogram x is given by ${p(\mathbf {x} \mid C_{k})={\frac {(\sum _{i}x_{i})!}{\prod _{i}x_{i}!}}\prod _{i}{p_{ki}}^{x_{i}}}$.

    Then, for a non-occurring feature i, the $x_i$ should be zero. It means that $x_i!$ = 1, and ${p_{ki}}^{x_i} = 1$. The non-occurring featurs does not contribute to $p(\mathbf {x} \mid C_{k})$.

To sum up, the different distribution between Bernoulli and Multinomial is the reason.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.