2
$\begingroup$

I am doing text classification but I am confused which Naive Bayes model I should use. What I understood by reading answers from couple of places that Gaussian Naive Bayes can be used if the attribute values are continuous, when attribute values are binary, binomial Naive Bayes can be used, for examples if we have words as features, we look into each sample to see if that word is present or not and thats how we get a matrix of S (sample) * V(vocubulary of words) dimension for text classification. Now, if we had actual word counts for creating S * V matrix, we would use multinomial Naive Bayes. My question is, if we use tf-idf (which has continuous/fraction value) for S * V matrix, which Naive Bayes Classification model should we use?

Am I getting conceptually wrong idea of data distribution?

$\endgroup$
  • $\begingroup$ Sounds to me like you are confusing the concept of attribute and class. The NB models are wrt. the class the model predicts, not the attributes used to do the prediction. $\endgroup$ – fnl Mar 22 '17 at 18:36
  • $\begingroup$ Did not get your point. The basic assumption in NB is given class labels the attributes are independent, isn't it? $\endgroup$ – MiNdFrEaK Mar 22 '17 at 22:37
  • $\begingroup$ Sure; But that's beside the point. A multinomial/Multinoulli NB model predicts one of $n$ classes where $n > 2$. A binomial/Bernoulli NB predicts for $n = 2$. And a normal/Gaussian NB gives you a real value instead of one choice of a discrete set of classes. So its all about the class(es), not the attributes; Check out the SciKit-Learn tutorial on NB if you still feel confused, that should help you get started. $\endgroup$ – fnl Mar 23 '17 at 10:36
  • $\begingroup$ BTW, if you like, think of Gaussian NB just as $n = 1$ (me ducks from bricks being thrown my way by hardliners) $\endgroup$ – fnl Mar 23 '17 at 10:42
2
$\begingroup$

Typically the multinomial naive bayes model would still be used - basically using the decimal TF-IDF values for each term in each document in place of the count for that term and proceeding as you usually would (TF-IDF is always $>= 0$). This paper provides details of one way to do that and study results:

Rennie, J.; Shih, L.; Teevan, J.; Karger, D. (2003). Tackling the poor assumptions of Naive Bayes classifiers (ICML.http://people.csail.mit.edu/jrennie/papers/icml03-nb.pdf).

Detailed explanation:

The idea is instead of using the term frequencies divided by the total number of terms as the categorical probabilities, you compute the TF-IDF representation of each document and use the fraction of TF-IDF values given to each term for a given class - i.e., sum up the values for a term across all documents in the class divided by the total of the sum of values for all terms - to get the probability value for each term. So basically TF-IDF value totals as opposed to count totals are used - instead of adding up 1s and 0s now you are adding decimal values, but the procedure is the same.

Then as with the traditional multinomial naive bayes you take the log of each term probability to get the log-linear decision function. However, in the traditional model you would then multiply this log value with the term frequency and sum across terms. Instead this paper proposes a final normalization first to normalize the log values across terms, before then applying the same linear decision rule.

Table 4 in the paper spells out this procedure clearly.

$\endgroup$
0
$\begingroup$

In naive bayes models for classification problems we are learning the distribution $P(X|C)\propto P(C) P(X|C)$ or more directly $P(C)\prod_i P(X_i|C)$, which is different from its counterpart in the generative discriminative pair: the logistic regression.

enter image description here

The grey circles represent manifest/observed variables.

If the $P(X|C)$'s are binary variables the model is a binomial Naive Bayes, and multinomial Naive Bayes if multinomial distribution and Gaussian if continuous Gaussian distribution(parameters are only mean and variance).

In case where x's are TF-IDF values I agree with the previous answer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.