Handling unknown words in language modeling tasks using LSTM

Question

For a natural language processing (NLP) task one often uses word2vec vectors as an embedding for the words. However, there may be many unknown words that are not captured by the word2vec vectors simply because these words are not seen often enough in the training data (many implementations use a minimum count before adding a word to the vocabulary). This may especially be the case with text from e.g. Twitter, where words are often misspelled.

How should such unknown words be handled when modeling a NLP task such as sentiment prediction using a long short-term (LSTM) network? I see two options:

1. Adding an 'unknown word' token to the word2vec dictionary.
2. Deleting these unknown words such that the LSTM doesn't even know the word was in the sentence.

What is the preferred way of handling these words?

Answer 1

Option 1 (adding an unknown word token) is how most people solve this problem.

Option 2 (deleting the unknown words) is a bad idea because it transforms the sentence in a way that is not consistent with how the LSTM was trained.

Another option that has recently been developed is to create a word embedding on-the-fly for each word using a convolutional neural network or a separate LSTM that processes the characters of each word one at a time. Using this technique your model will never encounter a word that it can't create an embedding for.

Answer 2

Mapping rare words to simply means that we delete those words and replace them with the token in the training data. Thus our model does not know of any rare words. It is a crude form of smoothing because the model assumes that the token will never actually occur in real data or better yet it ignores these n-grams altogether.