I am working on a classification problem and have built an LSTM network (character-level RNN). It has been trained and performs well. Now, when predicting, there are sometimes characters that are not in the original alphabet the model was trained on. What is the best practice to treat these characters? Should we ignore them ? I tried this and did not seem to work well. DO we need to retrain the model to also have an unknown character?
You will almost always have unknown words/characters in your test data e.g. unique names in sentences. For example a sentence in training data might be 'Thank you John' and in test data be 'Thank you Mary'. One way to overcome this issue in training phase is to create a dummy unknown token apart from all the tokens of your training data. For example if using a word embedding like fasttext, you could insert a new token with a embedding vector of all zeros/random initialised. This way, the above sequence will be 'Thank you '.
Since the unknown token is not included in your vocabulary while training and you need it while testing you should retrain it after adding it into the that vocabulary. It is impossible to consider all characters which may encounter always adding the unknown token into the vocabulary is a good practice.