I am creating an n-gram model that will predict the next word after an n-gram (probably unigram, bigram and trigram) as coursework.

I have seen lots of explanations about HOW to deal with zero probabilities for when an n-gram within the test data was not found in the training data. I understand how 'add-one' smoothing and some other techniques work.

However, I can find nothing about WHY we need to take actions such as these.

For instance, if the test data has "Peace begins with a Smile" and this was not in the training data, so when I supply the model with "Peace begins with a", it will not come up with "Smile" end word. It may provide others or none. If there are none or they have a low probability, then I would supply the shorter n-gram of "begins with a" and see what words and probabilities that provides. If that fails, then "with a" and so on.

I suspect I'm missing something but can't see what.

Please can you help?

  • $\begingroup$ What you call a low probability is not a problem as such, though it is an indicator that your prediction is likely to be wrong. The problem is what you call none: the shorter phrase does not appear in your training data, and in the worst case perhaps the preceding word does not appear at all in your training data. $\endgroup$
    – Henry
    Aug 4, 2019 at 9:39
  • $\begingroup$ Thanks @Henry. I understand that, but I’m not sure how any of the smoothing techniques help. In the end, it simply isn’t in the training data so can’t be reliably predicted. $\endgroup$
    – Chris
    Aug 4, 2019 at 13:44

1 Answer 1


What you are saying is called interpolation or back-off which is another technique that handles zero probability of n-grams.

See and check this: http://www.cs.cornell.edu/courses/cs4740/2014sp/lectures/smoothing+backoff.pdf


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.