1
$\begingroup$

I have read on the advantages of using character-level language models over word-level ones. In particular, you don't have to deal with the problem of out of vocabulary (OOV) words, since characters can generate any word as long as your vocabulary encompasses all the characters required for your task. But in this case, what do you do with OOV characters?

I'm training a simple char-RNN for text generation using tensorflow. My training set is in English, thus, I just chose the set of characters 'printable' from the python 'string' module as my vocabulary. Each character is mapped to an index and an embedding layer learns the representations of each character. This is mostly fine, however, some rare words appear with non-English accents, resulting in OOV characters, such as "é" or "ö". Also, even if I delete these instances from my training set, I would like my application to be robust such that if I later include an OOV character as a seed text to generate new text it won't break. So, even if I were to read the whole corpus once at the start to find all possible characters during training, what would I do if the model is given an OOV character for inference later?

What's the best practice to deal with OOV characters in this case? Map to an 'unknown' character index? Replace with a blank space? It is implied that these characters are rare and I don't know a prior how many of them there are, otherwise I would just include them all in the vocabulary from the start.

$\endgroup$
1
  • $\begingroup$ On this site there's no need to say "thank you" at the end of your post - it might seem rude at first, but it's part of the philosophy of this site to "Ask questions, get answers, no distractions", and it means future readers of your question don't need to read through the pleasantries. $\endgroup$ Dec 5, 2020 at 17:23

1 Answer 1

1
$\begingroup$

Having a special unknown token is a common practice regardless of whether you work with words or characters.

However, note that the model needs to learn how to handle the unknown characters. If any unknown character never appears in the training data, its embedding will end up randomly initialized and the model will behave randomly if such a character appears at inference time. So what you want to do is either exclude some infrequent characters from the training data from the vocabulary or sample the unknown symbols randomly at the places of some infrequent characters.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.