For a natural language processing (NLP) task one often uses word2vec vectors as an embedding for the words. However, there may be many unknown words that are not captured by the word2vec vectors simply because these words are not seen often enough in the training data (many implementations use a minimum count before adding a word to the vocabulary). This may especially be the case with text from e.g. Twitter, where words are often misspelled.
How should such unknown words be handled when modeling a NLP task such as sentiment prediction using a long short-term (LSTM) network? I see two options:
- Adding an 'unknown word' token to the word2vec dictionary.
- Deleting these unknown words such that the LSTM doesn't even know the word was in the sentence.
What is the preferred way of handling these words?