In my current language model my model is unaware of any token that is out-of-bag
for example:- In my summary generating model when we pass some token that is out-of-bag then my model will completely forget that token entirely. Such token might be any Noun such as name of a person, city or anything. And when my main token of the text is unknown in this case John who is main character of any story whose summary we are going to extract, how can i ensure that John is also include in my model although it is out-of-bag vocabulary. Summary not including John will make summary very poor.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
The standard approach is to encode the rare words in your dataset using a special token, UNK by convention, so that any new, out-of-vocabulary word would be labelled as belonging to the "rare word" category. By doing this, we expect the model to learn how to deal with the "other" words. You can find more sophisticated solutions in the machine translation literature (you don't want UNK's in the output of Google Translate), but for vast majority of problems doing this would be enough.
