# Handling unknown words in language modeling tasks using LSTM

For a natural language processing (NLP) task one often uses word2vec vectors as an embedding for the words. However, there may be many unknown words that are not captured by the word2vec vectors simply because these words are not seen often enough in the training data (many implementations use a minimum count before adding a word to the vocabulary). This may especially be the case with text from e.g. Twitter, where words are often misspelled.

How should such unknown words be handled when modeling a NLP task such as sentiment prediction using a long short-term (LSTM) network? I see two options:

1. Adding an 'unknown word' token to the word2vec dictionary.
2. Deleting these unknown words such that the LSTM doesn't even know the word was in the sentence.

What is the preferred way of handling these words?

• I've answered a similar question earlier; while the question then was not specific to LSTMs, it seems most of what I wrote there would be just as applicable: stats.stackexchange.com/questions/163005/… – fnl Mar 21 '16 at 9:11

Option 1 (adding an unknown word token) is how most people solve this problem.

Option 2 (deleting the unknown words) is a bad idea because it transforms the sentence in a way that is not consistent with how the LSTM was trained.

Another option that has recently been developed is to create a word embedding on-the-fly for each word using a convolutional neural network or a separate LSTM that processes the characters of each word one at a time. Using this technique your model will never encounter a word that it can't create an embedding for.

• Hi Aaron, Can you give me a couple pointers (papers or code) that use your third option? – Prophecies Jul 10 '17 at 18:34
• arxiv.org/abs/1508.02096 Here's one – Aaron Jul 11 '17 at 2:20
• – Fanglin Jan 17 '18 at 21:19
• One more recent one (EMNLP 2017) arxiv.org/abs/1707.06961 with code github.com/yuvalpinter/Mimick – jayelm Apr 17 '18 at 3:54
• But what is the rule of thumb for deciding what is the min_freq before a word is counted as unknown/made up <ukn>? Is this a function of the number of parameters of the actual model doing the learning task (e.g. a function of the number of parameters for the transfromer)? – Charlie Parker Mar 23 at 17:45

Mapping rare words to simply means that we delete those words and replace them with the token in the training data. Thus our model does not know of any rare words. It is a crude form of smoothing because the model assumes that the token will never actually occur in real data or better yet it ignores these n-grams altogether.

• Please add substantially to this answer. E.g. back up the claim that "adding an unknown word token is the best option". – Jim Jul 12 '18 at 11:49