I understood that we need to apply for Laplace smoothing to the words that are not present in our training data. But then why/what is the need to do Laplace smoothing for all the words (even the words that are present in the training set)?


2 Answers 2


Laplace smoothing is about adding a constant to all the counts and normalizing it.

$$ \hat p_i = \frac{n_i+\alpha}{\sum_{j=1}^K n_j+\alpha} $$

You can think of it as of adding a pseudocount $\alpha$ to each observed count $n_i$. So you observed $n_i$ words like this, but you transform it to $n_i+\alpha$.

Now imagine that you applied it only to the cases where the count was zero. In such cases, you would pretend that the count is $\alpha$. This could lead to a paradoxical situation where the words observed less than $\alpha$ times would be considered less frequent than the ones that you didn't observe at all. Only transforming all the counts preserves the relative frequencies. You are smoothing all the values instead of making up data.


Words that are not present in the training data are called Out of Vocabulary (OOV) words. You cannot apply laplace smoothing to words that are not in your vocabulary. You apply laplace smoothing to words that are in your vocabulary. This is because some vocabulary words that appear in one class may not appear in the other class. The maximum likelihood probabilities for the word in that other class would become 0. The following example illustrates the point.

Suppose you are doing sentiment analysis for positive or negative movie comments. Suppose your training set consists of one positive and one negative comment

+: "I like this alot"
-: "I don't like this"

You construct your vocabulary as ["I","like","this","alot","don't"] and you compute the respective maximum likelihood probabilities for each word belonging to each sentiment $P(w_i|+), P(w_i|-)$ where $w_i$ is a word in your vocabulary.

Suppose that in your test set, you encounter the comment "I don't think this is good". Now, we haven't seen the words "is" and "good", so we discard them away. Then we are left with the words "I don't think this".

To carry out Naive Bayes classification, we have to compute the posterior probabilities of the sentence belonging to either the positive or negative class.

$P(-|\text{"I don't think this"}) \propto P(\text{"I"}|-)P(\text{"don't"}|-)P(\text{think"}|-)P(\text{"this"}|-)P(-)$.

$P(+|\text{"I don't think this"}) \propto P(\text{"I"}|+)P(\text{"don't"}|+)P(\text{think"}|+)P(\text{"this"}|+)P(+)$.

Notice that if you don't use laplace smoothing, $P(\text{"don't"}|+)$ will be 0, because "don't" does not appear in a positive comment. Therefore, we apply laplace smoothing to give some probability mass to words in our vocabulary that may not have been seen in a positive comment, but may have been seen in a negative comment.

  • $\begingroup$ check this thread: here we are not always droping the OOV words. it is one of the strategy. link The question is that we are applying laplace smoothing to train set, even if the words are present in both the classes. @calveeen your example here has assumed that the word "don't" is not present in the positive class. $\endgroup$ Commented Mar 15, 2021 at 12:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.