33
$\begingroup$

I've dealt with Naive Bayes classifier before. I've been reading about Multinomial Naive Bayes lately.

Also Posterior Probability = (Prior * Likelihood)/(Evidence).

The only prime difference (while programming these classifiers) I found between Naive Bayes & Multinomial Naive Bayes is that

Multinomial Naive Bayes calculates likelihood to be count of an word/token (random variable) and Naive Bayes calculates likelihood to be following:

enter image description here

Correct me if I'm wrong!

$\endgroup$
49
$\begingroup$

The general term Naive Bayes refers the the strong independence assumptions in the model, rather than the particular distribution of each feature. A Naive Bayes model assumes that each of the features it uses are conditionally independent of one another given some class. More formally, if I want to calculate the probability of observing features $f_1$ through $f_n$, given some class c, under the Naive Bayes assumption the following holds:

$$ p(f_1,..., f_n|c) = \prod_{i=1}^n p(f_i|c)$$

This means that when I want to use a Naive Bayes model to classify a new example, the posterior probability is much simpler to work with:

$$ p(c|f_1,...,f_n) \propto p(c)p(f_1|c)...p(f_n|c) $$

Of course these assumptions of independence are rarely true, which may explain why some have referred to the model as the "Idiot Bayes" model, but in practice Naive Bayes models have performed surprisingly well, even on complex tasks where it is clear that the strong independence assumptions are false.

Up to this point we have said nothing about the distribution of each feature. In other words, we have left $p(f_i|c)$ undefined. The term Multinomial Naive Bayes simply lets us know that each $p(f_i|c)$ is a multinomial distribution, rather than some other distribution. This works well for data which can easily be turned into counts, such as word counts in text.

The distribution you had been using with your Naive Bayes classifier is a Guassian p.d.f., so I guess you could call it a Guassian Naive Bayes classifier.

In summary, Naive Bayes classifier is a general term which refers to conditional independence of each of the features in the model, while Multinomial Naive Bayes classifier is a specific instance of a Naive Bayes classifier which uses a multinomial distribution for each of the features.

References:

Stuart J. Russell and Peter Norvig. 2003. Artificial Intelligence: A Modern Approach (2 ed.). Pearson Education. See p. 499 for reference to "idiot Bayes" as well as the general definition of the Naive Bayes model and its independence assumptions

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ @jlund3, Thanks for the nice explanation. How doe we incorporate the information of the distribution in our classifier? I mean how does the fomula p(c|f1,...,fn)∝p(c)p(f1|c)...p(fn|c) changes based on whether it is a Guassian distribution vs multimodal $\endgroup$ – David Jul 9 '15 at 19:52
  • $\begingroup$ Thanks for the brief explanation but I recommend the book (Stuart J. Russell and Peter Norvig. 2003. Artificial Intelligence: A Modern Approach (2 ed.)) referenced above for more knowledge about NB and Artificial Intelligence Techniques too.. $\endgroup$ – Mirani May 16 '16 at 19:02
  • $\begingroup$ counts of multinomial distribution are non-independent. see my question here: datascience.stackexchange.com/questions/32016/… $\endgroup$ – Hanan Shteingart May 23 '18 at 15:09
11
+50
$\begingroup$

In general, to train Naive Bayes for n-dimensional data, and k classes you need to estimate $P(x_i | c_j)$ for each $1 \leq i \leq n$, $1 \leq j \leq k$ . You can assume any probability distribution for any pair $(i,j)$ (although it's better to not assume discrete distribution for $P(x_i|c_{j_1})$ and continuous for $P(x_i | c_{j_2})$). You can have Gaussian distribution on one variable, Poisson on other and some discrete on yet another variable.

Multinomial Naive Bayes simply assumes multinomial distribution for all the pairs, which seem to be a reasonable assumption in some cases, i.e. for word counts in documents.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

Previous answers are correct. This answer attempts to explain how the mentioned expressions are related.

The OP gave a specific case of Gaussian Naive Bayes, but the general Naive Bayes can be derived, and then the Multinomial Naive Bayes (with analogies to the word count example suggested), as follows:

Bayes (not Naive)

By Bayes Theorem, for any variables $C_k$ and $\mathbf{x}$

$p(C_k \mid \mathbf{x}) = \frac{p(C_k) \ p(\mathbf{x} \mid C_k)}{p(\mathbf{x})} = \frac{p(C_k, \mathbf{x})}{p(\mathbf{x})}$

When $\mathbf{x}$ is an evidence vector $\langle x_1, \dots, x_n\rangle$ (for example proportion $x_i$ of each vocabulary word $i$ in a document), then $p(C_k, \mathbf{x})$ = $p(C_k, x_1, \dots, x_n)$ (the numerator, or probability of all observations in union with the outcome)

By the chain rule, this union can be expanded to: $p(C_k, x_1, \dots, x_n) = p(x_1 \mid x_2, \dots, x_n, C_k) \ p(x_2 \mid x_3, \dots, x_n, C_k) \dots p(x_{n-1} \mid x_n, C_k) \ p(x_n \mid C_k) \ p(C_k)$

Note that the order of the arguments to $p$ doesn't matter since they are events being unioned and therefore the function is commutative. Therefore the above is the same as $p(x_n) p(x_{n-1}\mid x_n) \dots p(C_k|x_1\dots x_{n})$ (and any variation).

Bayes Assumption (Naive)

The Bayes assumption is that $x_i$ are mutually independent of one another. This means that $p(x_i|x_j) = p(x_i)$ for any distinct $x_i$ and $x_j$ and furthermore, $p(x_i|\mathbf{\hat{x}}) = p(x_i)$ for any $\mathbf{\hat{x}} \subset \mathbf{x} \setminus x_i $.

Therefore, the above chain rule expansion simplifies to a product
$p(C_k, x_1, \dots, x_n) = p(C_k) \prod_{i=1}^n p(x_i \mid C_k)$

General Naive Bayes

Substituting this finding into the first definition yields $p(C_k \mid \mathbf{x}) = \frac{p(C_k)}{p(\mathbf{x})}\prod_{i=1}^n p(x_i \mid C_k)$

Bernoulli version

For a Bernoulli trial with two outcomes and probability $q$ for one of those outcomes, the probability of this outcome occurring exactly $k$ out of $m$ times is

$p(k, m)={m \choose k} q^k (1 - q)^{n-k}$

The analogy with the document/word example above, is having two possible words, one of them occurring with proportion $q$ in the corpus, and computing the probability that it occurs $k$ times in a document with $m$ words.

Multinomial version

You probably have more than two words in the vocabulary. Multinomial distribution is a just a generalization of the binomial distribution derived using a couple combinatorics concepts: Dixon's identity, and the number of ways to form a combination.

$\frac{m!}{x_1!\cdots x_n!}q_1^{x_1}\cdots q_n^{x_n}$ where $m = \sum\limits_{i=1}^{n}x_n$ (e.g. $m$ is number of words in observed document)

$p(\mathbf{x} \mid C_k) = \frac{(\sum_i x_i)!}{\prod_i x_i !} \prod_i {p_{ki}}^{x_i}$ $\implies p(C_k \mid \mathbf{x}) = \frac{p(C_k)}{p(\mathbf{x})} \frac{(\sum_i x_i)!}{\prod_i x_i !} \prod_i {p_{ki}}^{x_i}$ (from substituting into the general Naive Bayes equation above)

For practicality, you may only care about relative probabilities (take away the need to compute $p(\mathbf{x})$ and replace "$=$" with "$\varpropto$")

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.