To clarify the slightly ambiguous language in the title. I have an RNN (actually 2 stacked RNN layers) that take input X of size
X [batch_size, sequence_length, features]
the model is trying to use the sequence_length number of timesteps to predict a single value output y_hat of size
y_hat [batch_size]
My sequence_length is set to a fixed size, but I have some sequences bigger than this value. I have seen people solving this by splitting the sequence into chunks of size sequence_length and then passing them in one after the other with the RNN states initialized with the final state of the previous chunk. This I understand. What is not clear to me is what is the target (y) for all but the final chunk, how is error backpropagated, and how weights are updated.
If I give an example just to make this clear, in the case of a language model I may have an input like the following:
hello i am generating an example and our target is interest
If our max length was 5, then we'd split up the sequence into:
chunk 1: X_1 = hello i am generating an, y_1 = ??
chunk 2: X_2 = example and our target is, y_2 = interest
If both the first and the second chunk are passed the final input as y (interest) we'd effectively be training the network that interest is the next word in the first subsequence above, as well as in the second?