In a recurrent neural network, you would usually forward propagate through several time steps, "unroll" the network, and then back propagate across the sequence of inputs.
Why would you not just update the weights after each individual step in the sequence? (the equivalent of using a truncation length of 1, so there is nothing to unroll) This completely eliminates the vanishing gradient problem, greatly simplifies the algorithm, would probably reduce the chances of getting stuck in local minima, and most importantly seems to work fine. I trained a model this way to generate text and the results seemed comparable to results I have seen from BPTT trained models. I am only confused on this because every tutorial on RNNs I have seen says to use BPTT, almost as if it is required for proper learning, which is not the case.
My only guess for why it is normally propagated through time is that maybe it learns faster or tends to converge on a more accurate solution. I have not yet tried a more traditional RNN with BPTT to get a truly conclusive comparison, but I can't think of any reason why it would be better.
Update: Here is an in depth description of the algorithm without BPTT
I am using a LSTM. In the example image, the input at time t is the letter "w" (using one hot encoding) and the cellstate (c) and hidden state (h) that was calculated at time t-1. The target output is the letter "o" (the next letter in the training sequence).
You first forward propagate to get a prediction for what the next letter should be. Then you back propagate the error and update the weights. Then you move to time t+1 where the inputs are the letter "o" along with the cellstate and hidden state that were just calculated, and now the target is "r"
The forward and back propagation don't actually work with sequences directly, but still learns the temporal dependencies just like if you were forward/back propagating over several inputs in the training sequence all at once.
I am using cross entropy along with softmax (-Log(output) * label)
This is literally no different than the average LSTM, except the sequence length for forward/back prop is 1.
