Unknown word problem is dealt by purposefully plugging-in a specific word - “” in the training corpus.After plugging-in, the ngram models will contain probability of the word “” and thus when an unknow word is encountered in the test corpus, we can use the probability of “” as an estimation.
We choose the rule of plugging-in “” as follows: loop over the list of tokens from beginning to start, each time a new token appears, we replace it with “”. After the looping, we use the modified tokens to generate ngrams. For example we have the first sentence and after transformation, its tokens will be exactly like the sencond sentence:
1. "Aizen is a cat. Hirusaki is a cat too. Hirusaki is a" 2. "<UNK> <UNK> <UNK> <UNK> <UNK> <UNK> is a cat <UNK>. Hirusaki is a"
Now I want to find the probability of
Hirusaki is dog
dog is a word out of vocabulary, but
"Hirusaki is" is not out of vocabulary. How should I find probability, so that it doesnt get zero.
What I did is, I treated dog as
<unk> but it still gives me zero probability