Zero-counts in Kneser-Ney smoothing

Hope to get help from someone experienced with implementation of language models. I am trying to implement n-gram model based on Kneser-Ney smoothing, however met the next issue:

Here is simplified equation for 4-gram Knesey-Ney smoothing computation. The main purpose of smoothing and interpolation here is to involve into calculation of word's probability the lower levels of given ngram. The next part of equation: represents the weight for lower ngram in equation, however with high n-grams as trigrams or higher, the sequence of words in test data quite possible will not occur even once in train data. That will make the weight equal to zero and since, all calculations with lower n-grams don't have a sanse. Moreover, it will return a zero probability, even all words in a given ngram are presented in vocabulary.

So, my question is if I understand correctly the model and Kneser-Ney smoothing and how it's possible to manage with zero-counts of particular words' sequences in this case.

Thanks you so much!

• please fix your title – Glen_b Aug 10 '17 at 2:39
• done, any suggestions? – Ivan Telnov Aug 10 '17 at 6:46
• Not really my area, sorry – Glen_b Aug 10 '17 at 7:01
• What do you mean that all calculations with lower n-grams don't have a sense? Using the lower order n-grams is exactly how the smoothing is done. – Aaron Aug 13 '17 at 19:12
• as I ve understood, the equation in Kneser Ney smoothing should be solved recursively from the very low level of n-gram to the highest one. The probabilities of lower ngram then multiplied on lambda weight, which in fact with high-order ngrams as 4-gram will be equal to zero in case the exact SEQUANCE of words was unseen in traininng data. So finally, we will multiply weighted probabilities of lower ngrams on zero. – Ivan Telnov Aug 14 '17 at 8:34