2
$\begingroup$

I would like to improve on my recommendation system. Imagine I have training data of $M=7,000,000$ samples. Each training sample contains a variable number of words in the body, and a variable amount of tags assigned to the case. There are roughly 100 million unique words, and roughly 40,000 unique tags.

The goal is to, given a set of words, to correctly classify the tag(s) when the tags are not observed.

Intuition tells me to simply compute the probability for each word, and then use the conditional probability for each tag given the set of words for a tag. More concretely:

Imagine one case has 5 unique words. Each word will have a $P(W_i)$. This is a constant value for the training set, and can be stored in memory. The probability of $T_1$ given $P(W_1, W_2....W_5)$, is then simply the number of occurrences in the intersection of $T_1$ for each word divided by $M$.

Why does Naive Bayes suggest using (according to the Bayes rule) $\frac{P(W_1..W_5|T_1) * P(T_1)}{P(W_1...W_5)}$ when $P(T_1|W_1...W_5)$ seems easier to obtain, and more intuitive*?

* "What is the probability of a tag given some combination of words?" seems to be exactly what we want to know.

|cite|improve this question|||||
$\endgroup$
6
$\begingroup$

As the number of words that you condition $P(T_1|W_1,..,W_N)$ on increases, you will find that calculating this probability will no longer be easy. The more conditions you add, the sparser your conditional probability tables will get.

Naive Bayes makes the assumption that the probability of words occurrences are independent of each other conditioned on the tag, thereby reducing $P(W_1,...,W_N|T_1)$ to $P(W_1|T_1)*...*P(W_N|T_1)$. This 'naive' independence assumption results in:

$P(T_1|W_1,..,W_N) \propto P(W_1|T_1) * ... * P(W_N|T_1) * P(T_1) $

Note how the denominator: $P(W_1,...,W_N)$ is dropped since it only serves as a normalising constant that ensures the probability values you get are between $0$ and $1$. You do not really need it to calculate the $T_i$ that maximises the formula given above.

|cite|improve this answer|||||
$\endgroup$
  • 1
    $\begingroup$ I am still not entirely sure how it is different. Isn't it also possible to have the independence assumption in $P(T|W)$? $P(T_1|W_1) * P(T_1|W_2)$... etc. $\endgroup$ – PascalVKooten Dec 8 '13 at 23:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.