The main advantage of working with character-level generative models is that the discrete space you're working with is much smaller -- there are about 97 English-language characters in common usage if we include all punctuation marks. By contrast, a vocabulary is many thousands of words. This implies that just storing the word embeddings will require a lot of memory, and including word embeddings in a model adds many, many parameters to the model so the computational cost is much higher on this account than the character-level model.
Mispelled words or other words not appearing in the vocabulary are usually treated as a special "unknown" token. This suffices for some practical contexts, but it also means that the model is not terribly flexible when it comes to generate new text.
On the other hand, character-level models can spontaneously generate unusual words with some (small) probability. As you point out, this can also mean that nonsense words or obvious typos can creep into the generated text, but in my experience, a well-trained generative model usually does pretty well at learning how to spell.
One of the key papers in character-level RNNs is "Generating Sequences With Recurrent Neural Networks" by Alex Graves.