What is the Probability Distribution of NLTK Naive Bayes?

Question

As I know Naïve Bayes has various distributions, as said in Sci-kit learn manual:

The different naive Bayes classifiers differ mainly by the assumptions they make regarding the distribution of $P(x_i \mid y)$.

So, there are various methods of Naïve Bayes like, Gaussian NB, Multinomial Naïve Bayes, and I found Sci-kit learn has different libraries for them.

There is another famous Naïve Bayes library in Python named NLTK. I was trying to find out the distribution of NLTK based Naïve Bayes. After searching I felt:

1. It is not Multinomial (see here, and here)

2. But in the documentation, I am finding the following examples, Example 6.4, Example6.6, it seems NLTK’s Naïve Bayes Classifier is supporting, nltk.FreqDist.

So I am bit confused. If any one of the esteemed members may kindly guide me.

Answer 1

I think that the probability distribution of NLTK's NB is Bernoulli. In the source, the classifier does keep a word frequency count, but don't forget that you are feeding this classifier a feature set, which is data type tuple, with two elements, dictionary (features) and string (label), when training. Since you are feeding it a dictionary, that means each word only appears once, so I don't see how the classifier can use the frequency of words in its calculations.

Check out this literature on Naive Bayes. You can use the previous button to navigate to the page where they talk about the multinomial NB model. Try recreating their example using NLTK and you will find that it classifies d5 as 'japan' instead of 'china' (which is the case in the Bernoulli variant) because NLTK omits the duplicate occurrences of 'chinese' which increases P(class = 'japan'|d5).

Here is my code for the example:

   ##Replicating example from Stanford literature to show that NLTK Naive  Bayes is calculating its conditionals in a Bernoulli fashion 

   ##The example can be found here: http://nlp.stanford.edu/IR-book      /html/htmledition/naive-bayes-text-classification-1.html

   ##They calculate the conditional probability for each label given d5 using both the multinomial model and Bernoulli model.

   ##When you run this code, the classifier labels d5 as 'japan', which is the case when you calculate the conditional prob in the Bernoulli model.


   import nltk
   from nltk.classify import NaiveBayesClassifier

   #Creating training and test data
   d1 = ['chinese', 'beijing', 'chinese']
   d2 = ['chinese', 'chinese', 'shanghai']
   d3 = ['chinese', 'macao']
   d4 = ['tokyo', 'japan', 'chinese']

   d5 = ['chinese', 'chinese', 'chinese', 'tokyo', 'japan']

   #Feature extractor
   def word_feats(words):
       return dict([(word, True) for word in words])

   #Feature sets
   d1_feats = [(word_feats(d1), 'china')]
   d2_feats = [(word_feats(d2), 'china')]
   d3_feats = [(word_feats(d3), 'china')]
   d4_feats = [(word_feats(d4), 'japan')]

   d5_feats = [(word_feats(d5), 'china')]

   #Training and Test feature sets
   train_feats = d1_feats + d2_feats + d3_feats + d4_feats
   test_feats = d5_feats

   cl = NaiveBayesClassifier.train(train_feats)

   #Creating list of probability distributions to extract probabilities from
   probDist = []
   for i in range(0, len(test_feats)):
       probdist = cl.prob_classify(test_feats[i][0])
       probDist.append(probdist)

   #Creating list of the max probabilities for prediction 
   prob = []
   for i in range(0, len(probDist)):
       prob.append(probDist[i].prob(probDist[i].max()))

   #Creating list of predicted labels for test data
   pred_labels = []
   for i in range(0, len(test_feats)):
       pred_labels.append(cl.classify(test_feats[i][0]))

   #classifying d5_feats
   cl.classify(word_feats(d5)) #returns 'japan' as predicted label