In many natural language processing applications such as spelling correction, machine translation and speech recognition, we use language models. Language models are created usually by counting how often sequences of words (n-grams) occur in a large corpus and normalizing the counts to create a probability. To account for unseen n-grams, we use smoothing methods (see several listed here) which take some of the probability mass from the n-grams which are attested in the model and distribute this mass among lower order n-gram (shorter word sequences) backoff probabilities.

Many of the smoothing techniques become mathematically complex because of the constraint that the calculations must keep the distribution as a probability (must add up to 1).

What is the reason for this constraint? What is the advantage of using strict probabilities for prediction instead of scores of any other kind?

P.S. The reference corresponding to the link is [Stanley F. Chen and Joshua Goodman (1998), “An Empirical Study of Smoothing Techniques for Language Modeling"].

  • 1
    $\begingroup$ I don't work in this field, but I don't see why summing the observed values & then dividing each value by the total should make the algorithm unwieldy. It seems to me that if the models are too complex, slow, or numerically unstable (etc.), the problem is most likely elsewhere. $\endgroup$ – gung - Reinstate Monica Jul 14 '12 at 18:22
  • $\begingroup$ No dividing out the counts in the first place is not so bad. It gets more complicated when you do smoothing. Katz, for example: en.wikipedia.org/wiki/Katz's_back-off_model $\endgroup$ – user9617 Jul 14 '12 at 23:36
  • $\begingroup$ @user9617 your link is dead could you please update it or better add the reference so people can still Google the resource in the future? Thanks in advance $\endgroup$ – Antoine Dec 1 '15 at 14:26
  • $\begingroup$ @Antoine done. I don't quite understand what happened to the PDF I was linking to before, but this one is just as good. $\endgroup$ – user9617 Dec 2 '15 at 2:54
  • $\begingroup$ @user9617 Thanks +1! I added the corresponding reference in case the link dies again in the future. $\endgroup$ – Antoine Dec 2 '15 at 9:09

The main advantages of using strict probabilities are a) ease of interpretation of the numbers; and b) being able to use Bayes theorem and other probabilistic methods in subsequent analysis. In some situations though, it is unnecessary. For example if you just want to rank the results with no further analysis, then there's no need to normalise the scores.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.