I want to use a neural network to do text analysis.

If I use a large dictionary, then it will contain all the words in training and test set, but the size of the dictionary is too large which will make the neural network training hard and time consuming. Another option is using the words in training set only, and make it a dictionary, for test set, all the words that are not in dictionary will be disregarded. Which option is better?

In general, I think for a bag of words model, the second option doesn't harm too much, and we may use it. But for other models, for example, if we treat the text as a series (time series), and we use a recurrent neural network, disregarding some words may break the relationship in this series or create a wrong relationship. So how to set the dictionary for the above situations?

  • 1
    $\begingroup$ This question would be more appropriately tagged with "word-embedding" and "bag-of-words". On the other hand, the tag "cross-validation" and "sentiment-analysis" have nothing to do with the question and should be removed. $\endgroup$ – fnl Jul 25 '15 at 15:46
  • $\begingroup$ What sort of text analysis: binary classification, n-class classification, sentiment analysis, topic labeling, ...? $\endgroup$ – smci Jul 21 '16 at 20:19

The standard method for handling unseen words is to use word embeddings, and in your particular case/question, neural embeddings, like the word2vec model by Mikolov. The principal, underlying approach is to add "unseen words" to your training set, so it does not depend on the machine learning method being used.

One way is to randomly pepper your training set with "unseen" words, replacing a random selection of your low-frequency words with "unseen" ones by replacing those words with the special token UNSEEN, which you add to your vocabulary ("dictionary") for this purpose. Another, even simpler approach is to replace all your words in the training set with a frequency below some cutoff (2 or 3, normally - i.e., words that only appear once or twice and as such are often quite "useless") with that special token UNSEEN.

Just in case this is not obvious: Do not use the actual string "UNSEEN", as it is likely to occur in your training or test data, or at least in production. Use some symbol that you are absolutely sure will remain a unique sentinel in your system, e.g., by padding it with special characters that would have been separated by your tokenizer.

Then, when you actually do find missing words during the test phase or in production, your system already has learned how to handle the situation gracefully and will still produce the best possible result, because you feed it this special UNSEEN token instead (of the actual, never-before-seen word).

Overall, the vocabulary for your system is the entire space of training words plus this "unseen word" UNSEEN (plus the padding tokens for the word vectors, naturally, but this is unrelated to the question). Thus, the ("feature extracted") WV you then get from your inputs can mitigate the strong effect unseen words normally have in your downstream processing, because all the "seen" words (during training) in the surrounding of the unseen word (during testing/production) will aid in figuring out the semantic space of this unseen word. Overall, this is probably half of the explanation why word embeddings are so popular (the other half being the semantic aspect).

An example of this, using RNNs and Theano, can be found in this deep learning tutorial, mostly based on the paper linked to earlier.

It should be noted that this approach of handling unseen words is not restricted to neural networks/embeddings, and computationally far more efficient methods like Hellinger PCA or low-rank multi-view learning (LR-MVL) can achieve the same effect with equal performance.

To make this discussion complete, it is necessary to mention two more advanced techniques for this issue. One is to include the orthographic properties in the learning process, which can be helpful (yet expensive, because you have to match words based on similarity) when words are just misspelled and can help detect words that are proper nouns or symbols ("1a"), because they have rather particular orthography. This approach was, for example, suggested in an ArXiv paper (Note that they wrongly use the word morphological here, which is actually something very different to the orthographic properties they are measuring; well, this is ArXiv, not JMLR...) Finally, the most advanced technique maybe is to forgo words entirely and train you neural network right off the character stream.

EDIT 26/06/2017: As this answer is still getting votes, an update might be a good idea. My latest favorite way of handling unseen words is by also learning embeddings for character n-grams ("subword information"), as implemented by FastText and described in an (warning: not peer reviewed) arXiv document entitled "Enriching Word Vectors with Subword Information".

  • $\begingroup$ My neural network is RNN. And I only use word embedding as an initial value, and still train the wordvec during the training phase. And I don't quite understand the method you mentioned above: If I don't know the test set in advance, how do I know which word that will be "unseen", and if I know the test set, can I simply train the wordvec for train and test sets together? $\endgroup$ – Hao Yu Jul 24 '15 at 16:40
  • $\begingroup$ fnl, I appreciated if you can explain more or give some links. $\endgroup$ – Hao Yu Jul 25 '15 at 1:32
  • $\begingroup$ Yes, but you input to whatever you use downstream, either a RNN, CRF, or whatever classifier, are word vectors (WV) aka. "word embeddings". And the vocabulary for your (neural) word embedding system simply is the entire space of training words plus the "unseen word". The ("feature extracted") WV you then get from your inputs can mitigate the strong effect unseen words normally have in your downstream processing. In the end, this is why word embeddings are so popular. When I have more time, I'll add exact references. $\endgroup$ – fnl Jul 25 '15 at 11:43
  • $\begingroup$ OK, I've updated the post to reflect the issue/open questions and linked to a simple tutorial putting the whole process into practice. Hope this clears things up, but let me know if not! $\endgroup$ – fnl Jul 25 '15 at 14:01
  • $\begingroup$ Those are cool methods! But what is the rule of thumb for deciding what is the min_freq before a word is counted as unknown/made up <ukn>? Is this a function of the number of parameters of the actual model doing the learning task (e.g. a function of the number of parameters for the transfromer)? $\endgroup$ – Charlie Parker Mar 23 at 17:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.