I'm trying to implement a system for text categorization using Naive Bayes as part of a school project. I have to hand code the algorithm and have been having some issues.
To make sure I understand exactly how Naive Bayes works, I decided to make a very simple example and manually perform Naive Bayes on it to make sure I get the right result.
So suppose there are two categories, DISASTER and POLITICS. There are only two training documents, one labelled DISASTER and one labelled POLITICS. (So the prior probabilities are both 0.5 and can be discarded in the calculations).
There is a single test document which we wish to classify as belonging to either DISASTER or POLITICS.
The documents are all very short, and consist of the following words: TRAINING (DISASTER): disaster disaster disaster disaster clown TRAINING (POLITICS): disaster clown politics politics politics... politics (40 consecutive occurrences of politics)
TEST document: map disaster politics politics politics
Intuition: Since the test document has so many occurrences of politics, I conjecture that it should be classified as POLITICS.
But if I use Naive Bayes', I get the following results (I used a simple smoothing technique, where I assumed that if a word was not seen before, I just gave it a count of 1 instead of 0):
P(test document|category = DISASTER) = (1/5)(4/5)(1/5)(1/5)(1/5) P(test document|category = POLITICS) = (1/42)(1/42)(40/42)(40/42)(40/42)
These calculations show that P(doc|DISASTER)>P(doc|POLITICS) and hence Naive Bayes would classify this test document as belonging to DISASTER, which makes no sense to me. Obviously I have made a mistake somewhere, or am not doing it correctly. What's the issue?