The standard method for handling unseen words is to use word embeddings, and in your
particular case/question, neural embeddings, like the word2vec model by Mikolov. The principal, underlying approach is to add "unseen words" to your training set, so it does not depend on the machine learning method being used.
One way is to randomly pepper your training set with "unseen" words, replacing a random selection of your low-frequency words with "unseen" ones by replacing those words with the special token UNSEEN
, which you add to your vocabulary ("dictionary") for this purpose. Another, even simpler approach is to replace all your words in the training set with a frequency below some cutoff (2 or 3, normally - i.e., words that only appear once or twice and as such are often quite "useless") with that special token UNSEEN
.
Just in case this is not obvious: Do not use the actual string "UNSEEN", as it is likely to occur in your training or test data, or at least in production. Use some symbol that you are absolutely sure will remain a unique sentinel in your system, e.g., by padding it with special characters that would have been separated by your tokenizer.
Then, when you actually do find missing words during the test phase or in production, your system already has learned how to handle the situation gracefully and will still produce the best possible result, because you feed it this special UNSEEN
token instead (of the actual, never-before-seen word).
Overall, the vocabulary for your system is the entire space of training words plus this "unseen word" UNSEEN
(plus the padding tokens for the word vectors, naturally, but this is unrelated to the question). Thus, the ("feature extracted") WV you then get from your inputs can mitigate the strong effect unseen words normally have in your downstream processing, because all the "seen" words (during training) in the surrounding of the unseen word (during testing/production) will aid in figuring out the semantic space of this unseen word. Overall, this is probably half of the explanation why word embeddings are so popular (the other half being the semantic aspect).
An example of this, using RNNs and Theano, can be found in this deep learning tutorial, mostly based on the paper linked to earlier.
It should be noted that this approach of handling unseen words is not restricted to neural networks/embeddings, and computationally far more efficient methods like Hellinger PCA or low-rank multi-view learning (LR-MVL) can achieve the same effect with equal performance.
To make this discussion complete, it is necessary to mention two more advanced techniques for this issue. One is to include the orthographic properties in the learning process, which can be helpful (yet expensive, because you have to match words based on similarity) when words are just misspelled and can help detect words that are proper nouns or symbols ("1a"), because they have rather particular orthography. This approach was, for example, suggested in an ArXiv paper (Note that they wrongly use the word morphological here, which is actually something very different to the orthographic properties they are measuring; well, this is ArXiv, not JMLR...) Finally, the most advanced technique maybe is to forgo words entirely and train you neural network right off the character stream.
EDIT 26/06/2017: As this answer is still getting votes, an update might be a good idea. My latest favorite way of handling unseen words is by also learning embeddings for character n-grams ("subword information"), as implemented by FastText and described in an (warning: not peer reviewed) arXiv document entitled "Enriching Word Vectors with Subword Information".