I am working through an example of Add-1 smoothing in the context of NLP

Say that there is the following corpus (start and end tokens included)

+ I am sam -
+ sam I am -
+ I do not like green eggs and ham -

I want to check the probability that the following sentence is in that small corpus, using bigrams

+ I am sam green -

Normally, the probability would be found by:


Which would be:

(2/3)*(2/3)*(1/2)*(0/2)*(0/1) = 0

To try to alleviate this, I would do the following:


Where V is the sum of the types in the searched sentence as they exist in the corpus, in this instance:


This turns out to be:

(3/14)*(3/14)*(2/13)*(1/13)*(1/12)~= 0.000045

Now, say I want to see the probability that the following sentence is in the small corpus:

+ I am mark Johnson -

A normal probability will be undefined (0/0).


Going straight to the smoothing portion:

V = Count(+)+Count(i)+Count(am)+Count(mark)+Count(johnson)=3+3+2+0+0=8


I fail to understand how this can be the case, considering "mark" and "johnson" are not even present in the corpus to begin with. Is this a special case that must be accounted for? Or is this just a caveat to the add-1/laplace smoothing method?

Do I just have the wrong value for V (i.e. should I add 1 for a non-present word, which would make V=10 to account for "mark" and "johnson")? If this is the case (it almost makes sense to me that this would be the case), then would it be the following:


Moreover, what would be done with, say, a sentence like:

+ yo soy mark johnson -

Would it be (assuming that I just add the word to the corpus):


Usually, n-gram language model use a fixed vocabulary that you decide on ahead of time. In the smoothing, you do use one for the count of all the unobserved words. It's possible to encounter a word that you have never seen before like in your example when you trained on English but now are evaluating on a Spanish sentence. The out of vocabulary words can be replaced with an unknown word token that has some small probability. This is consistent with the assumption that based on your English training data you are unlikely to see any Spanish text.

Another thing people do is to define the vocabulary equal to all the words in the training data that occur at least twice. The words that occur only once are replaced with an unknown word token. This way you can get some probability estimates for how often you will encounter an unknown word.

  • $\begingroup$ If I am understanding you, when I add an unknown word, I want to give it a very small probability. Essentially, V+=1 would probably be too generous? $\endgroup$ – basil Oct 7 '16 at 23:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.