# Capturing initial patterns when using truncated backpropagation through time (RNN/LSTM)

Say that I use an RNN/LSTM to do sentiment analysis, which is a many-to-one approach (see this blog). The network is trained through a truncated backpropagation through time (BPTT), where the network is unrolled for only 30 last steps as usual.

In my case each of my text sections that I want to classify are much longer than the 30 steps being unrolled (~100 words). Based on my knowledge BPTT is only run a single time for a single text section, which is when it has passed over the entire text section and computed the binary classification target, $y$, which it then compares to the loss function to find the error.

The gradients will then never be computed with regards to the first words of each text section. How can the RNN/LSTM then still adjust its weights to capture specific patterns that only occur within the first few words? For instance, say that all sentences marked as $positive$ start with "I love this" and all sentences marked as $negative$ start with "I hate this". How would the RNN/LSTM capture that when it is only unrolled for the last 30 steps when it hits the end of a 100-step long sequence?

It's true that limiting your gradient propagation to 30 time steps will prevent it from learning everything possible in your dataset. However, it depends strongly on your dataset whether that will prevent it from learning important things about the features in your model!

Limiting the gradient during training is more like limiting the window over which your model can assimilate input features and hidden state with high confidence. Because at test time you apply your model to the entire input sequence, it will still be able to incorporate information about all of the input features into its hidden state. It might not know exactly how to preserve that information until it makes its final prediction for the sentence, but there might be some (admittedly weaker) connections that it would still be able to make.

Think first about a contrived example. Suppose your network is to generate a 1 if there is a 1 anywhere in its input, and a 0 otherwise. Say you train the network on sequences of length 20 and limit then gradient to 10 steps. If the training dataset never contains a 1 in the final 10 steps of an input, then the network is going to have a problem with test inputs of any configuration. However, if the training set has some examples like [1 0 0 ... 0 0 0] and others like [0 0 0 ... 1 0 0], then the network will be able to pick up on the "presence of a 1" feature anywhere in its input.

Back to sentiment analysis then. Let's say during training your model encounters a long negative sentence like "I hate this because ... around and around" with, say, 50 words in the ellipsis. By limiting the gradient propagation to 30 time steps, the model will not connect the "I hate this because" to the output label, so it won't pick up on "I", "hate", or "this" from this training example. But it will pick up on the words that are within 30 time steps from the end of the sentence. If your training set contains other examples that contain those same words, possibly along with "hate" then it has a chance of picking up on the link between "hate" and the negative sentiment label. Also, if you have shorter training examples, say, "We hate this because it's terrible!" then your model will be able to connect the "hate" and "this" features to the target label. If you have enough of these training examples, then the model ought to be able to learn the connection effectively.

At test time, let's say you present the model with another long sentence like "I hate this because ... on the gecko!" The model's input will start out with "I hate this", which will be passed into the hidden state of the model in some form. This hidden state is used to influence future hidden states of the model, so even though there might be 50 words before the end of the sentence, the hidden state from those initial words has a theoretical chance of influencing the output, even though it was never trained on samples that contained such a large distance between the "I hate this" and the end of the sentence.

@Imjohns3 has right, if you process long sequences(size N) and limit backpropagation to last K steps, the network won't learn patterns at the begining.

I have worked with long texts and use the approach where I compute loss and do backpropagation after every K steps. Let's assume that my sequence had N=1000 tokens, my RNN process first K=100 then I try to do prediction (compute loss) and backpropagate. Next while maintaining the RNN state brake the gradient chain(in pytorch->detach) and start another k=100 steps.

A good example of this technique you can find here: https://github.com/ksopyla/pytorch_neural_networks/blob/master/RNN/lstm_imdb_tbptt.py