Hope to get help from someone experienced with implementation of language models. I am trying to implement n-gram model based on Kneser-Ney smoothing, however met the next issue:

Here is simplified equation for 4-gram Knesey-Ney smoothing computation.

enter image description here

The main purpose of smoothing and interpolation here is to involve into calculation of word's probability the lower levels of given ngram. The next part of equation:

Weight for lower ngram

represents the weight for lower ngram in equation, however with high n-grams as trigrams or higher, the sequence of words in test data quite possible will not occur even once in train data. That will make the weight equal to zero and since, all calculations with lower n-grams don't have a sanse. Moreover, it will return a zero probability, even all words in a given ngram are presented in vocabulary.

So, my question is if I understand correctly the model and Kneser-Ney smoothing and how it's possible to manage with zero-counts of particular words' sequences in this case.

Thanks you so much!

  • $\begingroup$ please fix your title $\endgroup$ – Glen_b Aug 10 '17 at 2:39
  • $\begingroup$ done, any suggestions? $\endgroup$ – Ivan Telnov Aug 10 '17 at 6:46
  • $\begingroup$ Not really my area, sorry $\endgroup$ – Glen_b Aug 10 '17 at 7:01
  • $\begingroup$ What do you mean that all calculations with lower n-grams don't have a sense? Using the lower order n-grams is exactly how the smoothing is done. $\endgroup$ – Aaron Aug 13 '17 at 19:12
  • $\begingroup$ as I ve understood, the equation in Kneser Ney smoothing should be solved recursively from the very low level of n-gram to the highest one. The probabilities of lower ngram then multiplied on lambda weight, which in fact with high-order ngrams as 4-gram will be equal to zero in case the exact SEQUANCE of words was unseen in traininng data. So finally, we will multiply weighted probabilities of lower ngrams on zero. $\endgroup$ – Ivan Telnov Aug 14 '17 at 8:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.