I understood that we need to apply for Laplace smoothing to the words that are not present in our training data. But then why/what is the need to do Laplace smoothing for all the words (even the words that are present in the training set)?


Words that are not present in the training data are called Out of Vocabulary (OOV) words. You cannot apply laplace smoothing to words that are not in your vocabulary. You apply laplace smoothing to words that are in your vocabulary. This is because some vocabulary words that appear in one class may not appear in the other class. The maximum likelihood probabilities for the word in that other class would become 0. The following example illustrates the point.

Suppose you are doing sentiment analysis for positive or negative movie comments. Suppose your training set consists of one positive and one negative comment

+: "I like this alot"
-: "I don't like this"

You construct your vocabulary as ["I","like","this","alot","don't"] and you compute the respective maximum likelihood probabilities for each word belonging to each sentiment $P(w_i|+), P(w_i|-)$ where $w_i$ is a word in your vocabulary.

Suppose that in your test set, you encounter the comment "I don't think this is good". Now, we haven't seen the words "is" and "good", so we discard them away. Then we are left with the words "I don't think this".

To carry out Naive Bayes classification, we have to compute the posterior probabilities of the sentence belonging to either the positive or negative class.

$P(-|\text{"I don't think this"}) \propto P(\text{"I"}|-)P(\text{"don't"}|-)P(\text{think"}|-)P(\text{"this"}|-)P(-)$.

$P(+|\text{"I don't think this"}) \propto P(\text{"I"}|+)P(\text{"don't"}|+)P(\text{think"}|+)P(\text{"this"}|+)P(+)$.

Notice that if you don't use laplace smoothing, $P(\text{"don't"}|+)$ will be 0, because "don't" does not appear in a positive comment. Therefore, we apply laplace smoothing to give some probability mass to words in our vocabulary that may not have been seen in a positive comment, but may have been seen in a negative comment.

  • $\begingroup$ check this thread: here we are not always droping the OOV words. it is one of the strategy. link The question is that we are applying laplace smoothing to train set, even if the words are present in both the classes. @calveeen your example here has assumed that the word "don't" is not present in the positive class. $\endgroup$ Mar 15 at 12:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.