What would be the approach needed for creating a text generator. Most of the tutorials such
follow using characters instead of words. Could someone tell me what are the tradeoffs using words vs characters. I believe characters will reduce the size of the vectors. However, wouldnt it yield to better results?
I believe characters will reduce the size of the vectors.
Well it depends what features you will use. For sure if you use one-hot encoding then there are only a couple dozen characters, but you won't usually see people using one-hot encoded words, since the dimensionality of such vectors is huge. What is commonly used for words is called word embeddings.
Could someone tell me what are the tradeoffs using words vs characters.
Apart from the fact, that you can use one-hot vectors for characters in practice, the other difference is that character-level models handle unknown words better - these language models can infer something about unseen words based on patterns in known words.
For more approaches (for example hybrid approaches that use subword features) and for references see Goldberg's Primer on Neural Network Models for Natural Language Processing section 5.5.5.
You can absolutely use both words or characters. But if you cannot segment the text into words accurately it is better to use characters. However if the text is relatively too long I'd recommend words provided a sound segmentor is available since too long text may lead to bad network performance. If the vocabulary is too large when you use words you can use large corpus to pretrain the embedding employing a task such as language model(corpus can be numerously large).
