For a natural language processing (NLP) task one often uses word2vec vectors as an embedding for the words. However, there may be many unknown words that are not captured by the word2vec vectors simply because these words are not seen often enough in the training data (many implementations use a minimum count before adding a word to the vocabulary). This may especially be the case with text from e.g. Twitter, where words are often misspelled.

How should such unknown words be handled when modeling a NLP task such as sentiment prediction using a long short-term (LSTM) network? I see two options:

  1. Adding an 'unknown word' token to the word2vec dictionary.
  2. Deleting these unknown words such that the LSTM doesn't even know the word was in the sentence.

What is the preferred way of handling these words?

  • 2
    $\begingroup$ I've answered a similar question earlier; while the question then was not specific to LSTMs, it seems most of what I wrote there would be just as applicable: stats.stackexchange.com/questions/163005/… $\endgroup$
    – fnl
    Commented Mar 21, 2016 at 9:11

2 Answers 2


Option 1 (adding an unknown word token) is how most people solve this problem.

Option 2 (deleting the unknown words) is a bad idea because it transforms the sentence in a way that is not consistent with how the LSTM was trained.

Another option that has recently been developed is to create a word embedding on-the-fly for each word using a convolutional neural network or a separate LSTM that processes the characters of each word one at a time. Using this technique your model will never encounter a word that it can't create an embedding for.

  • $\begingroup$ Hi Aaron, Can you give me a couple pointers (papers or code) that use your third option? $\endgroup$
    – Prophecies
    Commented Jul 10, 2017 at 18:34
  • $\begingroup$ arxiv.org/abs/1508.02096 Here's one $\endgroup$
    – Aaron
    Commented Jul 11, 2017 at 2:20
  • $\begingroup$ code: github.com/wlin12/JNN $\endgroup$
    – Fanglin
    Commented Jan 17, 2018 at 21:19
  • 1
    $\begingroup$ One more recent one (EMNLP 2017) arxiv.org/abs/1707.06961 with code github.com/yuvalpinter/Mimick $\endgroup$
    – jayelm
    Commented Apr 17, 2018 at 3:54
  • $\begingroup$ But what is the rule of thumb for deciding what is the min_freq before a word is counted as unknown/made up <ukn>? Is this a function of the number of parameters of the actual model doing the learning task (e.g. a function of the number of parameters for the transfromer)? $\endgroup$ Commented Mar 23, 2021 at 17:45

Mapping rare words to simply means that we delete those words and replace them with the token in the training data. Thus our model does not know of any rare words. It is a crude form of smoothing because the model assumes that the token will never actually occur in real data or better yet it ignores these n-grams altogether.

  • 4
    $\begingroup$ Please add substantially to this answer. E.g. back up the claim that "adding an unknown word token is the best option". $\endgroup$
    – Jim
    Commented Jul 12, 2018 at 11:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.