I am wondering which combinations of Naive models can be paired with different vectorizing methods so that it makes sense.

Let's say we have a simple binary spam-classification task.

Multinomial model:

If I understand correctly, the Multinomial model is used to calculate the class-conditional probabilities based on the term frequencies. Let's say our input text consists of only 1 word for simplicity. So, the class-conditional probability (or likelihood) would be calculated as follows:

\begin{equation} \hat{P}(x_i \mid \omega_j) = \frac{tf_{xi, y} + \alpha}{tf_y + \alpha \cdot V} \end{equation}

  • $tf_{xi, y}$ = term frequency; the number of times the word $x_i$ occurs in the training dataset for samples of class $\omega_j$
  • $tf_y$ Sum of all term frequencies in the training dataset for class $\omega_j$
  • $\alpha$ is the addititive smoothening parameter ($\alpha = 1$ for Laplace smoothening
  • $V$ is the size of the vocabulary

Multi-variate Bernoulli (Binomial) model:

Here, we would use document frequencies rather than term frequencies since the distribution used binary values: "does a word occur in a document or not?"

\begin{equation} \hat{P}(x_i \mid \omega_j) = \frac{df_{xi, y} + 1}{df_y + 2} \end{equation}

  • $df_{xi, y}$ is the number of documents in the training dataset that contain $x_i$ and belong to class $\omega_j$
  • $df_y$ Number of documents in the training dataset that belong to class $\omega_j$
  • +1 and +2 are the parameters for Laplace smoothening


So, my question is which vectorizer do I have to use when I want to prepare the dataset for either of the models.

Let's say I am using the scikit-learn package in Python. from

from sklearn.feature_extraction.text import TfidfVectorizer,CountVectorizer
sklearn.naive_bayes import MultinomialNB
from sklearn.naive_bayes import BernoulliNB
from sklearn.pipeline import Pipeline

When I understand correctly, the count vectorizer produces a "bag of words" and for the term frequencies, so this combination seems to make sense to me:

clf_1 = Pipeline([
    ('vectorizer', CountVectorizer(ngram_range=(1,2), stop_words="english")),
    ('classifier',  MultinomialNB()),

But in terms of the document frequencies, is it okay to use CountVectorizer in combination with the Bernoulli model? Can this automatically retrieve the correct document frequencies from the count vectorizer results?

clf_2 = Pipeline([
    ('vectorizer', CountVectorizer(stop_words="english")),
    ('classifier', BernoulliNB()),

And how would I use the Tf-idf vectorizer? Does this combination make sense?

clf_3 = Pipeline([
    ('vectorizer',  TfidfVectorizer(ngram_range=(1,2), stop_words="english")),
    ('classifier',  MultinomialNB()),

I assume that it doesn't make sense to use TfidfVectorizer with the Bernoulli model, right?

  • $\begingroup$ You are correct that it is not right to use tfidf with naive bayes. $\endgroup$ – Aaron Apr 21 '15 at 17:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy