Measuring the contribution of word to a classification in a Naive Bayes document classifier

Question

I have a multinomial Naive Bayes document classifier. I'm interested in knowing the contribution made by each word to a single classification.

That is, I'd like to be able to measure which words in a document are contributing most to the classification.

Is this just:

$ P( Class\ |\ word) \ =\ \frac{P( word\ |\ Class) \ P( Class)}{P( word)} $

...?

And then perhaps normalize to 0.0 - 1.0 between the words in the document. Or are there other approaches that are used to understand a classification with this model?

To give an example, if I'm classifying documents into "Mentions People" (True) or not (False), then for the document "Bob did a thing" I'd perhaps see:

• "Bob" = 0.8
• "did" = 0.1
• "a" = 0.01
• "thing" = 0.09

...signifying the model was mainly using "Bob" to classify the document.

Answer 1

Consider the case of two classes and two words:

$ P( Class1 | word1, word2) \approx \frac{P( word1 | Class1) P(word2 | Class1) P(Class1)}{P( word1 | Class1) P(word2 | Class1) P(Class1) + P( word1 | Class2) P(word2 | Class2) P(Class2)} $

and

$ P( Class2 | word1, word2) \approx \frac{P( word1 | Class2) P(word2 | Class2) P(Class2)}{P( word1 | Class1) P(word2 | Class1) P(Class1) + P( word1 | Class2) P(word2 | Class2) P(Class2)} $

Note that the "evidence" portion (the denominator) is the same in each case, and includes terms for both classes. This will complicate your calculation, since you will need to "extract" the word for which you're trying to estimate the contribution.

Handling the numerator is simpler. Assuming you are interested in the contribution of word1 to the posterior probability for your document being in Class1 you simply take:

$ P(word1|Class1) $

Now for the denominator, note that you want to end up with:

$ P(word2 | Class1) P(Class1) + P(word2 | Class2) P(Class2) $

Why? Because that represents the denominator you would have, were word1 not part of your document.

So, you can then multiply your numerator by the original denominator, and then divide by the new denominator:

$ \frac{P( word1 | Class1)(P( word1 | Class1) P(word2 | Class1) P(Class1) + P( word1 | Class2) P(word2 | Class2) P(Class2))}{P(word2 | Class1) P(Class1) + P(word2 | Class2) P(Class2)} $