Is there an intuition why character based models language bases models are preferred over word based. For example Karpathy builds his language model by predicting the next character in Karpathy Blog.

The aspect I am struggling with is that not each combination of characters is a word, so intuitively I would try to predict the next word (or word embedding and calculate squared error). I think this is also used in the sentence embedding proposed by Kiros Skip Thought.

So my question is what are advantages and disadvantages for character based language models in comparison to word based models.

  • $\begingroup$ There are much less characters than words, resulting in smaller input space. $\endgroup$ – rcpinto Jun 3 '16 at 6:58
  • $\begingroup$ The answer I posted on Data Science: datascience.stackexchange.com/a/13179/22012 might be what you need. $\endgroup$ – minerals Mar 8 '17 at 21:11

The main advantage of working with character-level generative models is that the discrete space you're working with is much smaller -- there are about 97 English-language characters in common usage if we include all punctuation marks. By contrast, a vocabulary is many thousands of words. This implies that just storing the word embeddings will require a lot of memory, and including word embeddings in a model adds many, many parameters to the model so the computational cost is much higher on this account than the character-level model.

Mispelled words or other words not appearing in the vocabulary are usually treated as a special "unknown" token. This suffices for some practical contexts, but it also means that the model is not terribly flexible when it comes to generate new text.

On the other hand, character-level models can spontaneously generate unusual words with some (small) probability. As you point out, this can also mean that nonsense words or obvious typos can creep into the generated text, but in my experience, a well-trained generative model usually does pretty well at learning how to spell.

One of the key papers in character-level RNNs is "Generating Sequences With Recurrent Neural Networks" by Alex Graves.

| cite | improve this answer | |
  • $\begingroup$ I understand your point about character model taking much less space, but then on the other hand I would expect character model to have much more hidden layers and much more parameters to be able to perform as well as word level model. This I guess would also require lots of memory and computational cost no? $\endgroup$ – jeremie Nov 15 '19 at 12:36
  • $\begingroup$ A network can be of any size, so I guess it depends on what you want to do and how large a network you need to achieve your desired level of effectiveness. I've trained effective character-level models using 1-hot inputs, and 2 layers of LSTMs with 512 units each. The parameter count is pretty low; by contrast, pre-trained Glove embeddings by themselves consume more memory than this whole model. $\endgroup$ – Sycorax Nov 15 '19 at 13:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.