29
$\begingroup$

I've dealt with Naive Bayes classifier before. I've been reading about Multinomial Naive Bayes lately.

Also Posterior Probability = (Prior * Likelihood)/(Evidence).

The only prime difference (while programming these classifiers) I found between Naive Bayes & Multinomial Naive Bayes is that

Multinomial Naive Bayes calculates likelihood to be count of an word/token (random variable) and Naive Bayes calculates likelihood to be following:

enter image description here

Correct me if I'm wrong!

$\endgroup$
43
$\begingroup$

The general term Naive Bayes refers the the strong independence assumptions in the model, rather than the particular distribution of each feature. A Naive Bayes model assumes that each of the features it uses are conditionally independent of one another given some class. More formally, if I want to calculate the probability of observing features $f_1$ through $f_n$, given some class c, under the Naive Bayes assumption the following holds:

$$ p(f_1,..., f_n|c) = \prod_{i=1}^n p(f_i|c)$$

This means that when I want to use a Naive Bayes model to classify a new example, the posterior probability is much simpler to work with:

$$ p(c|f_1,...,f_n) \propto p(c)p(f_1|c)...p(f_n|c) $$

Of course these assumptions of independence are rarely true, which may explain why some have referred to the model as the "Idiot Bayes" model, but in practice Naive Bayes models have performed surprisingly well, even on complex tasks where it is clear that the strong independence assumptions are false.

Up to this point we have said nothing about the distribution of each feature. In other words, we have left $p(f_i|c)$ undefined. The term Multinomial Naive Bayes simply lets us know that each $p(f_i|c)$ is a multinomial distribution, rather than some other distribution. This works well for data which can easily be turned into counts, such as word counts in text.

The distribution you had been using with your Naive Bayes classifier is a Guassian p.d.f., so I guess you could call it a Guassian Naive Bayes classifier.

In summary, Naive Bayes classifier is a general term which refers to conditional independence of each of the features in the model, while Multinomial Naive Bayes classifier is a specific instance of a Naive Bayes classifier which uses a multinomial distribution for each of the features.

References:

Stuart J. Russell and Peter Norvig. 2003. Artificial Intelligence: A Modern Approach (2 ed.). Pearson Education. See p. 499 for reference to "idiot Bayes" as well as the general definition of the Naive Bayes model and its independence assumptions

$\endgroup$
  • $\begingroup$ The links are broken $\endgroup$ – ssoler Jul 22 '14 at 14:05
  • $\begingroup$ @jlund3, Thanks for the nice explanation. How doe we incorporate the information of the distribution in our classifier? I mean how does the fomula p(c|f1,...,fn)∝p(c)p(f1|c)...p(fn|c) changes based on whether it is a Guassian distribution vs multimodal $\endgroup$ – David Jul 9 '15 at 19:52
  • $\begingroup$ Thanks for the brief explanation but I recommend the book (Stuart J. Russell and Peter Norvig. 2003. Artificial Intelligence: A Modern Approach (2 ed.)) referenced above for more knowledge about NB and Artificial Intelligence Techniques too.. $\endgroup$ – Mirani May 16 '16 at 19:02
  • $\begingroup$ counts of multinomial distribution are non-independent. see my question here: datascience.stackexchange.com/questions/32016/… $\endgroup$ – Hanan Shteingart May 23 '18 at 15:09
10
+50
$\begingroup$

In general, to train Naive Bayes for n-dimensional data, and k classes you need to estimate $P(x_i | c_j)$ for each $1 \leq i \leq n$, $1 \leq j \leq k$ . You can assume any probability distribution for any pair $(i,j)$ (although it's better to not assume discrete distribution for $P(x_i|c_{j_1})$ and continuous for $P(x_i | c_{j_2})$). You can have Gaussian distribution on one variable, Poisson on other and some discrete on yet another variable.

Multinomial Naive Bayes simply assumes multinomial distribution for all the pairs, which seem to be a reasonable assumption in some cases, i.e. for word counts in documents.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.