Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 5 down vote favorite

I am trying to understand naive Bayes and its application to text classification. I have a doubt or this may be my misconception.

Suppose we have two categories "News" and "Sports" in which we need to classify any given document. Let the dictionary contain only 3 keywords ${news,football,tennis}$ with the following parameters

\begin{align*} P(news/News)&=0.99,P(news/Sports)=0.01,P(tennis/Sports)=0.9,\\ P(tennis/News)&=0.1,P(football/Sports)=0.9,P(football/News)=0.1, \\ P(Sports)&=0.5 ,P(News)=0.5,P((news,football,tennis))=k; \end{align*}

We get a document which has all the three keywords. So when we evaluate \begin{align} P(News/(news,football,tennis))=0.99\cdot0.1\cdot0.1\cdot0.5/k=0.00495/k\\ P(Sports/(news,football,tennis))=0.01\cdot0.9\cdot0.9\cdot0.5/k=0.00405/k \end{align}

So the document is classified to "News" category, but intuitively we know that it should belong to "Sports" category.

share|improve this question
did you miss the conditioning variable in P(News|), P(Sports|) or it means P(.| All three keywords)? – suncoolsu Aug 9 '11 at 7:21
intuitively we know that it should belong to "Sports" category. Why? It seems to me given the numbers you chose that it is almost impossible to have the word news in Sports. So if the word news is in the document that would trump everything else and make the document highly likely to be News. – Muhammad Alkarouri Aug 9 '11 at 14:56
Yep, I think the problem is the numbers. Even when the numbers are derived from "real life" they can produce erroneous results for specific cases (ie, fail to predict "reality"), and when you pick the numbers out of the air you can get really bizarre results. In particular, you'll hardly ever see a probability of 0.99 in "real life". And I'm trying to get my head around whether there's a lack of independence here, though thinking about that always makes my head hurt. – Daniel R Hicks Aug 10 '11 at 0:47
Thinking about it, I suspect, in real life, there is some correlation between the terms that you'd intuitively take into account (which is part of why you find the results strange), but which the naive Bayesian equation doesn't. Ie, "news" negatively predicts (if that makes any sense) "football", but you use the same weights for "football" regardless of the presence of "news". – Daniel R Hicks Aug 10 '11 at 20:54

2 Answers

up vote 1 down vote

Well: naive Bayes is called naive for a reason: the assumed conditional independence is often doubtful, even though it turns out to work well in a lot of practical cases.

Besides that: you have "chosen" your conditional probabilities so that it turns out this way. There is no (a priori) reason why P(tennis|News) and P(tennis|Sports) should sum to 1, but in this case this leads to the counterintuitive results.

share|improve this answer
I think P(tennis|News) and P(tennis|Sports) should add upto 1. – Amit Aug 9 '11 at 7:52
1  
Why do you think so? – Nick Sabbe Aug 9 '11 at 8:03
Ouch. That is painfully wrong. What you are describing is P(News | tennis) and P(Sports|tennis). e.g.: the chance that it is News given that it contains "tennis" (this is indeed a given for the 10 documents). – Nick Sabbe Aug 9 '11 at 8:25
Yes my bad...got your point – Amit Aug 9 '11 at 8:46
up vote 1 down vote

A naive Bayes classifier, as the names suggests, is a simple application of Bayes' Theorem. Basically, it calculates the probabilities of quantities of interest (generally unobserved, called parameters or latent classes) based on the observed data. In your case the observed data are: news, football, and tennis. The quantities of interest for which you want to calculate the probabilities are: News and Sports. Seems like you are interested in calculating: $P(\text{News}|\text{news}, \text{football}, \text{tennis}), P(\text{News}|\text{news}, \text{football}, \text{tennis})$.

Now we will use Bayes theorem to get:

$$ P(\text{News}|\text{news}, \text{football}, \text{tennis}) = \frac{P(\text{news}, \text{football}, \text{tennis}|\text{News})P(\text{News})}{P(\text{news}, \text{football}, \text{tennis})} $$ The first term in the numerator is calculated using the fact that given you observe the latent class, that is, News, the observed data, that is news, football, and tennis probabilities are independent (this may be a questionable assumption, but the answer depends on subject matter). You can use the law for calculating the probabilties of independent event.
$$ P(\text{news}, \text{football}, \text{tennis}|\text{News})=P(\text{news}|\text{News})P( \text{football}|\text{News})P(\text{tennis}|\text{News}) $$

Proceeding similarly for Sports, we get:

$$ P(\text{Sports}|\text{news}, \text{football}, \text{tennis}) = \frac{P(\text{news}, \text{football}, \text{tennis}|\text{Sports})P(\text{Sports})}{P(\text{news}, \text{football}, \text{tennis})} $$ $$ P(\text{news}, \text{football}, \text{tennis}|\text{Sports})=P(\text{news}|\text{Sports})P( \text{football}|\text{Sports})P(\text{tennis}|\text{Sports}) $$

The denominator for both cases can be calculated by using the Law of total probability.

$$ P(\text{news}, \text{football}, \text{tennis}) =P(\text{news}, \text{football}, \text{tennis}|\text{News})P(\text{News})+ P(\text{news}, \text{football}, \text{tennis}|\text{Sports})P(\text{Sports}) $$

We are now left with only one probability in each case, that is,$P(\text{News})$ and $P(\text{Sports})$, respectively. If we know these, every probability until now can be calculated. This can be determined based on prior knowledge, or in your case it might be already provided to you.

Plugging in all the probabilities gives you the probabilities of interest.

A high probability value for a specific class implies that the observed document belongs to that class (News or Sports). But how do you decided "how high is high", depends, again, on subject matter and a lot of other issues.

share|improve this answer
thnx for you reply, but i think i have used the same formulas, i didn't get what you wanted to add – Amit Aug 9 '11 at 7:50

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.