I have read on the advantages of using character-level language models over word-level ones. In particular, you don't have to deal with the problem of out of vocabulary (OOV) words, since characters can generate any word as long as your vocabulary encompasses all the characters required for your task. But in this case, what do you do with OOV characters?
I'm training a simple char-RNN for text generation using tensorflow. My training set is in English, thus, I just chose the set of characters 'printable' from the python 'string' module as my vocabulary. Each character is mapped to an index and an embedding layer learns the representations of each character. This is mostly fine, however, some rare words appear with non-English accents, resulting in OOV characters, such as "é" or "ö". Also, even if I delete these instances from my training set, I would like my application to be robust such that if I later include an OOV character as a seed text to generate new text it won't break. So, even if I were to read the whole corpus once at the start to find all possible characters during training, what would I do if the model is given an OOV character for inference later?
What's the best practice to deal with OOV characters in this case? Map to an 'unknown' character index? Replace with a blank space? It is implied that these characters are rare and I don't know a prior how many of them there are, otherwise I would just include them all in the vocabulary from the start.
