How to train a RNN language model?

Question

I want to train a RNN-based language model from https://arxiv.org/pdf/1409.2329.pdf for next word prediction. How to split the sentences from the dataset into input and ground truth during the training?

Let's say I have the following training sample: The quick brown fox jumps over the lazy dog.

Does it makes sense to take every possible separation of this sentence? Doesn't this lead to overfitting?

input="The"                                     GT="quick"
input="The quick"                               GT="brown"
input="The quick brown"                         GT="fox"
...

Or can I just use only one last word as ground truth since the P(dog | The quick brown fox jumps over the lazy) already calculate probabilities of all previous words?

input="The quick brown fox jumps over the lazy" GT="dog"

Answer 1

The input is the ground truth.

In other words, typical discriminative supervised learning tries to learn $p(y|x)$. On the other hand, language models learn $p(x)$. As you can see there is no need for any labeled "$y$"

The way this works with autoregressive models such as RNNs is that you decompose it as a bunch of conditional probabilities $p(x) = \prod_j p(x_j|x_{<j})$.