1
$\begingroup$

I have noticed some ambiguity/inconsistency in how various authors calculate the p(word|label) estimate in Multinomial Naive Bayes. In some cases, this value is calculated by counting words without regard to which documents those words are in. Dan Jurafsky does so in his lecture notes Text Classification and Naïve Bayes and several related videos:

Parameter estimation

This widely-cited paper also calculates that value by counting words, without regard to document:

Tackling the Poor Assumptions of Naive Bayes Text Classifiers

Parameters must be estimated from the training data. We do this by selecting a Dirichlet prior and taking the expectation of the parameter with respect to the posterior. For details, we refer the reader to Section 2 of Heckerman (1995). This gives us a simple form for the estimate of the multinomial parameter, which involves the number of times word i appears in the documents in class c (Nci), divided by the total number of word occurrences in class c (Nc).

However, I have seen other sources that compute this value by counting documents, not words, such as these lecture notes from Andrew Ng:

CS229 Lecture notes

The parameters have a very natural interpretation. For instance, φj|y=1 is just the fraction of the spam (y = 1) emails in which word j does appear.

I have worked out various toy examples by hand, trying things both ways (as well as with/without Laplace smoothing, with the "naive" independence assumption holding/violated, and with all documents having the same/different number of words) and noted the following:

  • When all documents contain the same number of words, counting either way gives the same result.
  • When the "naive" assumption does hold true, counting either way gives the same result.
  • When documents contain different numbers of words and the "naive" assumption is actually not true (i.e. the typical case) you get different numbers depending on which way you count. Notably, you even get different numbers when calculating the values for "full" Bayes.

I have not been able to find any explanation for these two different ways of calculating the p(word|label) value, and quite frankly (pace Jurafsky) counting by word seems like a mistake to me. However, since the difference only seems to matter when the "naive" independence assumption is violated, I wonder if perhaps this is somehow meant to correct for that? If so, I have yet to see any explanation/justification for this.

Can anybody shed some light on why there seem to be two fundamentally different ways of calculating this value?

$\endgroup$
1
$\begingroup$

When you count documents, you assume that of word occurs at least once, this is enough for you to make decision. For example, if you see "Viagra" in some e-mail you can be relatively sure that it is a spam. You would also count documents if you don't expect specific words to appear multiple times in documents (i.e. when it wouldn't matter much).

It is the opposite with counting words, if more occurrences would make you more convinced to make some classification, then you would count them. Example may be an e-mail that mentions "lots of money" multiple times, if you wouldn't know anything else about it, then the significant count of such phrase might make you think that this may be some kind of scam.

In many machine learning applications people use both kinds of counts, by using TF-IDF encoding.

$\endgroup$
  • $\begingroup$ I’m not referring to Bernoulli vs Multinomial here though, which I believe you’re alluding to. I’ve seen people calculate p(word|label) by counting words, even in cases where each word appears only once per document. Jurafsky and others seem to suggest always counting by word, without justification or explanation (that I have seen.) Consider a dataset with one document “foo bar” and another “foo” both labeled “spam” — is p(foo|spam) = 2/2 (counting by documents) or 2/3 (counting by words) ? $\endgroup$ – Bill Clark May 15 at 21:29
  • $\begingroup$ Ah, this question seems more related to your point: stats.stackexchange.com/questions/57337/… $\endgroup$ – Bill Clark May 16 at 14:36
  • 1
    $\begingroup$ I didn't mean Bernoulli vs multinomial. I was trying to say that different ways of counting may be needed for different datasets. Sometimes one way may be more reasonable, or simply "work better", then other for some dataset. Saying it differently: it is not only on how to calculate probabilities, but also probabilities of what do you want to calculate. $\endgroup$ – Tim May 16 at 14:44
  • $\begingroup$ I'm looking for a theoretical justification here though, not a heuristic one. The issue that concerns me most is that when calculating p(word|label) by counting words, everybody (that I have seen) still calculates p(label) by counting documents -- which makes the total probability with non-naive Bayes sum to something other than 1. To me, that strongly implies that is the wrong way to count -- always. Sure, sticking other numbers into the calculations could end up producing better results, but without some kind of theoretical basis, we're really just trying things at random. $\endgroup$ – Bill Clark May 16 at 15:58
  • $\begingroup$ Ah, I just realized a mistake I made in calculating the posterior probabilities -- I'd not realized the independence assumption was conditional on the classes. This makes the posterior probabilities sum to 1 no matter what, basically. No matter which way you count. This still doesn't address my concerns re: counting two different ways and the theoretical basis for doing so, but it does alleviate my concerns regarding the sum being something other than 1. $\endgroup$ – Bill Clark May 16 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.