I want to train a RNN-based language model from https://arxiv.org/pdf/1409.2329.pdf for next word prediction. How to split the sentences from the dataset into input and ground truth during the training?

Let's say I have the following training sample: The quick brown fox jumps over the lazy dog.

Does it makes sense to take every possible separation of this sentence? Doesn't this lead to overfitting?

input="The"                                     GT="quick"
input="The quick"                               GT="brown"
input="The quick brown"                         GT="fox"

Or can I just use only one last word as ground truth since the P(dog | The quick brown fox jumps over the lazy) already calculate probabilities of all previous words?

input="The quick brown fox jumps over the lazy" GT="dog"

The input is the ground truth.

In other words, typical discriminative supervised learning tries to learn $p(y|x)$. On the other hand, language models learn $p(x)$. As you can see there is no need for any labeled "$y$"

The way this works with autoregressive models such as RNNs is that you decompose it as a bunch of conditional probabilities $p(x) = \prod_j p(x_j|x_{<j})$.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.