I have noticed some ambiguity/inconsistency in how various authors calculate the p(word|label) estimate in Multinomial Naive Bayes. In some cases, this value is calculated by counting words without regard to which documents those words are in. Dan Jurafsky does so in his lecture notes Text Classification and Naïve Bayes and several related videos:
This widely-cited paper also calculates that value by counting words, without regard to document:
Parameters must be estimated from the training data. We do this by selecting a Dirichlet prior and taking the expectation of the parameter with respect to the posterior. For details, we refer the reader to Section 2 of Heckerman (1995). This gives us a simple form for the estimate of the multinomial parameter, which involves the number of times word i appears in the documents in class c (Nci), divided by the total number of word occurrences in class c (Nc).
However, I have seen other sources that compute this value by counting documents, not words, such as these lecture notes from Andrew Ng:
The parameters have a very natural interpretation. For instance, φj|y=1 is just the fraction of the spam (y = 1) emails in which word j does appear.
I have worked out various toy examples by hand, trying things both ways (as well as with/without Laplace smoothing, with the "naive" independence assumption holding/violated, and with all documents having the same/different number of words) and noted the following:
- When all documents contain the same number of words, counting either way gives the same result.
- When the "naive" assumption does hold true, counting either way gives the same result.
- When documents contain different numbers of words and the "naive" assumption is actually not true (i.e. the typical case) you get different numbers depending on which way you count. Notably, you even get different numbers when calculating the values for "full" Bayes.
I have not been able to find any explanation for these two different ways of calculating the p(word|label) value, and quite frankly (pace Jurafsky) counting by word seems like a mistake to me. However, since the difference only seems to matter when the "naive" independence assumption is violated, I wonder if perhaps this is somehow meant to correct for that? If so, I have yet to see any explanation/justification for this.
Can anybody shed some light on why there seem to be two fundamentally different ways of calculating this value?