My goal is to generate text based on a specific book using an LSTM language model. One problem with my generator is that it seems that the book's dialogue is somewhat messing up my generator. I had the idea to separate dialogue/non-dialogue, as
- Dialogue/non-dialogue tend to use different words so it might improve the model's word usage to separate them rather than mixing them.
- I could get a correct structure for the dialogue (ie have the form "Text. <Dialogue> I said. Text"
When considering how to deal with this, one idea I had was to have a specific language model trained on the dialogue of the novel, and another trained on the non-dialogue. I would mark the non-dialogue data with a token. For example:
I walked forward. "Test dialogue" I said
Would be replaced with
I walked forward. <Dialogue> I said
Then, I would generate text by starting with the non-dialogue generator. Whenever a <Dialogue> token is reached, I would then generate with the dialogue generator. When the dialogue generator reaches its end (end token), I would continue with the non-dialogue generator.
This approach would give me a correct dialogue structure. However, there are a couple flaws I can think of:
- It takes extra effort to implement/train/tune two separate models.
- There is less training data, especially for the dialogue model.
- Since valid dialogue could be just one word, the variety in it might make it difficult to train directly.
- Because the dialogue is generated separately, it would not be influenced by the words before it. Losing this might make the dialogue not fit with the words appearing before.
One other idea I had was to mark the start and end of the dialogue each with a different token, and have that token have some kind of special property that could help create proper dialogue structure.
I would appreciate any ideas with how to deal with these flaws, or a different approach to this problem entirely.
