0
$\begingroup$

Unknown word problem is dealt by purposefully plugging-in a specific word - “” in the training corpus.After plugging-in, the ngram models will contain probability of the word “” and thus when an unknow word is encountered in the test corpus, we can use the probability of “” as an estimation.

We choose the rule of plugging-in “” as follows: loop over the list of tokens from beginning to start, each time a new token appears, we replace it with “”. After the looping, we use the modified tokens to generate ngrams. For example we have the first sentence and after transformation, its tokens will be exactly like the sencond sentence:

1. "Aizen is a cat. Hirusaki is a cat too. Hirusaki is a"

2. "<UNK> <UNK> <UNK> <UNK> <UNK> <UNK> is a cat <UNK>. Hirusaki is a"

Now I want to find the probability of Hirusaki is dog

So, dog is a word out of vocabulary, but "Hirusaki is" is not out of vocabulary. How should I find probability, so that it doesnt get zero.

What I did is, I treated dog as <unk> but it still gives me zero probability

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You can use Laplace smoothing, then for each <unk> you would assign same, small probability. What follows, for any <unk> in the text, you'd get same probability for the sentence <unk> is a dog.

If you want to be more precise, you need to have bigger dictionary, so that "Hirusaki" is no more unknown.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.