After lemmatization of text I have a sequence of sets of lemmas, because every word can correspond to more than one lemma. How should I extract ngram statistics based on that? The only thing that comes to mind is counting every possible combination, but this seems a bit weird. Is there any other option?

More implementation-related question: is there a way to do this by NLTK?


If you have uncertainty about which lemma each word belongs to, the best thing is to use that in your statistics. For instance you encounter the trigram "I walk around"and you have two possible lexemes for "walk", you should split that single occurence into two halves and increment the trigram frequencies of "I/1 walk/1 around/1" and "I/1 walk/2 around/1" by 0.5 each.

Even better, if you have a some distribution which gives one lexeme more probability than the other, you can use this probability to increment the frequencies.

In the end, you will probably use these frequencies to estimate the probability p(around | I, walk). To translate this to the space of lexemes, you must again sum over all possible lexemes for these words: $$ p(\mbox{around} \mid \mbox{I}, \mbox{walk}) = \sum_{i,j,k} p(\mbox{around}_i \mid \mbox{I}_j, \mbox{walk}_k) $$

and then estimate the probabilities in the sum with the statistics you've gathered.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.