# Language modeling: why is adding up to 1 so important?

In many natural language processing applications such as spelling correction, machine translation and speech recognition, we use language models. Language models are created usually by counting how often sequences of words (n-grams) occur in a large corpus and normalizing the counts to create a probability. To account for unseen n-grams, we use smoothing methods (see several listed here) which take some of the probability mass from the n-grams which are attested in the model and distribute this mass among lower order n-gram (shorter word sequences) backoff probabilities.

Many of the smoothing techniques become mathematically complex because of the constraint that the calculations must keep the distribution as a probability (must add up to 1).

What is the reason for this constraint? What is the advantage of using strict probabilities for prediction instead of scores of any other kind?

P.S. The reference corresponding to the link is [Stanley F. Chen and Joshua Goodman (1998), “An Empirical Study of Smoothing Techniques for Language Modeling"].

• I don't work in this field, but I don't see why summing the observed values & then dividing each value by the total should make the algorithm unwieldy. It seems to me that if the models are too complex, slow, or numerically unstable (etc.), the problem is most likely elsewhere. Jul 14, 2012 at 18:22
• No dividing out the counts in the first place is not so bad. It gets more complicated when you do smoothing. Katz, for example: en.wikipedia.org/wiki/Katz's_back-off_model Jul 14, 2012 at 23:36
• @user9617 your link is dead could you please update it or better add the reference so people can still Google the resource in the future? Thanks in advance Dec 1, 2015 at 14:26
• @Antoine done. I don't quite understand what happened to the PDF I was linking to before, but this one is just as good. Dec 2, 2015 at 2:54
• @user9617 Thanks +1! I added the corresponding reference in case the link dies again in the future. Dec 2, 2015 at 9:09