So far, I have seen two ways of writing multinomial NB, I was wondering which would be the correct one to use in theory?


Suppose we are going to classify the sentence

We are going really really fast


In terms of the likelihood, the two methods are described as the follows

  1. $P(we, are, going, really, really, fast|C_k) \\= P(we|C_k) P(are|C_k) P(going|C_k)P(really|C_k) P(really|C_k) P(fast|C_k)$

  2. $P(we, are, going, really, really, fast|C_k) \\= P(we=1,are=1,going=1,really=2,fast=1|C_k) \\= \frac{6!}{2!} P(we|C_k) P(are|C_k) P(going|C_k)P(really|C_k)^2 P(fast|C_k)$


The difference is whether it has the coefficient item of multinomial distribution. The coefficient measures the order effects.

In method one, the order matters, since we are not considering permutations of words, we are interested in only one particular word combination (the natural order).

However, for the second method, the order doesn't matter. We are counting the word occurrences, any permutations satisfy the counts would be taken into consideration.

I am confused as they seem like to be the same method, but missing the coefficient made them like two distinct methods. How should I understand such difference?

  • $\begingroup$ It depends what kind of features you'll provide for the algorithm (counts vs any occurrence). $\endgroup$ – Tim Jun 25 '18 at 7:27
  • $\begingroup$ Which one satisfy multinomial distribution, counts or occurrence or both? Counts make much sense to me to be categorized as multinomial distribution, you have events which are the counts and some probability associated with that event. Regarding the occurrence, I'm not sure what is probability distribution of a series of words (the indication of chain multiplication). $\endgroup$ – Logan Jun 25 '18 at 8:02
  • $\begingroup$ Your question seems to be about this coefficient 6!/2!... but I wonder about what is $P(x|C_k)$ ? why can you multiply these terms without taking into account correlations? Why do you not use en.wikipedia.org/wiki/Chain_rule_(probability) ? $\endgroup$ – Martijn Weterings Jun 25 '18 at 20:35
  • $\begingroup$ NB has strong independence assumptions between features. $\endgroup$ – Logan Jun 26 '18 at 1:41
  • $\begingroup$ You've already pointed out that they are two different methods, as in one, the order matters, and in the other, it doesn't. So why are you confused "as they seem like to be the same method"? You already know they aren't, and why not! $\endgroup$ – jbowman Jul 2 '18 at 2:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.