9
$\begingroup$

For a natural language processing (NLP) task one often uses word2vec vectors as an embedding for the words. However, there may be many unknown words that are not captured by the word2vec vectors simply because these words are not seen often enough in the training data (many implementations use a minimum count before adding a word to the vocabulary). This may especially be the case with text from e.g. Twitter, where words are often misspelled.

How should such unknown words be handled when modeling a NLP task such as sentiment prediction using a long short-term (LSTM) network? I see two options:

  1. Adding an 'unknown word' token to the word2vec dictionary.
  2. Deleting these unknown words such that the LSTM doesn't even know the word was in the sentence.

What is the preferred way of handling these words?

$\endgroup$
  • 2
    $\begingroup$ I've answered a similar question earlier; while the question then was not specific to LSTMs, it seems most of what I wrote there would be just as applicable: stats.stackexchange.com/questions/163005/… $\endgroup$ – fnl Mar 21 '16 at 9:11
9
$\begingroup$

Option 1 (adding an unknown word token) is how most people solve this problem.

Option 2 (deleting the unknown words) is a bad idea because it transforms the sentence in a way that is not consistent with how the LSTM was trained.

Another option that has recently been developed is to create a word embedding on-the-fly for each word using a convolutional neural network or a separate LSTM that processes the characters of each word one at a time. Using this technique your model will never encounter a word that it can't create an embedding for.

$\endgroup$
-1
$\begingroup$

Mapping rare words to simply means that we delete those words and replace them with the token in the training data. Thus our model does not know of any rare words. It is a crude form of smoothing because the model assumes that the token will never actually occur in real data or better yet it ignores these n-grams altogether.

$\endgroup$

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

  • 4
    $\begingroup$ Please add substantially to this answer. E.g. back up the claim that "adding an unknown word token is the best option". $\endgroup$ – Jim Jul 12 '18 at 11:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.