3
$\begingroup$

I am working on a classification problem and have built an LSTM network (character-level RNN). It has been trained and performs well. Now, when predicting, there are sometimes characters that are not in the original alphabet the model was trained on. What is the best practice to treat these characters? Should we ignore them ? I tried this and did not seem to work well. DO we need to retrain the model to also have an unknown character?

$\endgroup$
  • $\begingroup$ I have a similar problem: I trained the net to recognize 4 different patterns, but I don't know how to handle an unknown one, since it is classified anyway as one of the 4, even with a high score (> 0.9) $\endgroup$ – rookie coder Jul 27 '18 at 9:34
1
$\begingroup$

You will almost always have unknown words/characters in your test data e.g. unique names in sentences. For example a sentence in training data might be 'Thank you John' and in test data be 'Thank you Mary'. One way to overcome this issue in training phase is to create a dummy unknown token apart from all the tokens of your training data. For example if using a word embedding like fasttext, you could insert a new token with a embedding vector of all zeros/random initialised. This way, the above sequence will be 'Thank you '.

$\endgroup$
0
$\begingroup$

Since the unknown token is not included in your vocabulary while training and you need it while testing you should retrain it after adding it into the that vocabulary. It is impossible to consider all characters which may encounter always adding the unknown token into the vocabulary is a good practice.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.