# Why back propagate through time in a RNN?

In a recurrent neural network, you would usually forward propagate through several time steps, "unroll" the network, and then back propagate across the sequence of inputs.

Why would you not just update the weights after each individual step in the sequence? (the equivalent of using a truncation length of 1, so there is nothing to unroll) This completely eliminates the vanishing gradient problem, greatly simplifies the algorithm, would probably reduce the chances of getting stuck in local minima, and most importantly seems to work fine. I trained a model this way to generate text and the results seemed comparable to results I have seen from BPTT trained models. I am only confused on this because every tutorial on RNNs I have seen says to use BPTT, almost as if it is required for proper learning, which is not the case.

• An interesting direction to take this research would be to compare the results that you've achieved on your problem with benchmarks published in the literature on standard RNN problems. That would make a really cool article. – Sycorax Oct 10 '16 at 19:40
• Your "Update: I added an answer" replaced the previous edit with your architecture description and an illustration. Is it on purpose? – amoeba Oct 12 '16 at 23:01
• Yes I took it out because it didn't seem relevant to the actual question really and it took up a lot of space, but I can add it back if it helps – Frobot Oct 12 '16 at 23:08
• Well people seem to have massive problems with understanding your architecture, so I guess any additional explanations are useful. You can add it to your answer instead of your question, if you prefer. – amoeba Oct 13 '16 at 16:22

Edit: I made a big mistake when comparing the two methods and have to change my answer. It turns out the way I was doing it, just back propagating on the current time step, actually starts out learning faster. The quick updates learn the most basic patterns very quickly. But on a larger data set and with longer training time, BPTT does in fact come out on top. I was testing a small sample for just a few epochs and assumed whoever starts out winning the race will be the winner. But this did lead me to an interesting find. If you start out your training back propagating just a single time step, then change to BPTT and slowly increase how far back you propagate, you get faster convergence.

• Thank you for your update. In the source of that last image he says this about the one to one setting: "Vanilla mode of processing without RNN, from fixed-sized input to fixed-sized output (e.g. image classification)." So that's what we were saying. If it's as you've described it has no state and it's not an RNN. "forward propagating through a single input before back propagating" - I'd call that an ANN. But these wouldn't perform as good with text so something's up and I have no idea what because I don't have the code – ragulpr Oct 13 '16 at 0:18
• I didn't read that part and you are correct. The model I am using is actually the "many to many" on the far right. i assumed in the "one to one" section there were really many of these all connected and the drawing just left it out. but that actually is one of the options on the far right that i didn't notice (it's odd to have that one in there in a blog about RNNs, so i assumed they were all recurrent). I will edit that part of the answer to make more sense – Frobot Oct 13 '16 at 0:32
• I imagined that was the case, that's why I insisted on seeing your loss function. If it's many to many your loss is akin to $error=\sum_t(y_t-\hat{y}_t)^2$ and it's identically an RNN and you're propagating/inputing the whole sequence but then just truncating BPTT i.e you'd calculate the red part in my post but not recurse further. – ragulpr Oct 13 '16 at 0:51
• My loss function doesn't sum over time. I take one input, get one output, then calculate a loss, and update the weights, then move on to t+1, so there is nothing to sum. I will add the exact loss function to the original post – Frobot Oct 13 '16 at 0:58
• Just post your code I'm not doing any more guessing, this is silly. – ragulpr Oct 13 '16 at 1:00

"Unfolding through time" is simply an application of the chain rule, $$\frac{dF(g(x), h(x), m(x))}{dx} = \frac{\partial F}{\partial g}\frac{dg}{dx} + \frac{\partial F}{\partial h}\frac{dh}{dx} + \frac{\partial F}{\partial m}\frac{dm}{dx}$$

The output of an RNN at time step $t$, $H_t$ is a function of the parameters $\theta$, the input $x_t$ and the previous state, $H_{t-1}$ (note that instead $H_t$ may be transformed again at time step $t$ to obtain the output, that is not important here). Remember the goal of gradient descent: given some error function $L$, let's look at our error for the current example (or examples), and then let's adjust $\theta$ in such a way, that given the same example again, our error would be reduced.

How exactly did $\theta$ contribute to our current error? We took a weighted sum with our current input, $x_t$, so we'll need to backpropagate through the input to find $\nabla_\theta a(x_t, \theta)$, to work out how to adjust $\theta$. But our error was also the result of some contribution from $H_{t-1}$, which was also a function of $\theta$, right? So we need to find out $\nabla_\theta H_{t-1}$, which was a function of $x_{t-1}$, $\theta$ and $H_{t-2}$. But $H_{t-2}$ was also a function a function of $\theta$. And so on.

• I understand why you back propagate through time in a traditional RNN. I'm trying to find out why a traditional RNN uses multiple inputs at once for training, when using just one at a time is much simpler and also works – Frobot Oct 11 '16 at 2:46
• The only sense in which you can feed in multiple inputs at once into an RNN is feeding in multiple training examples, as part of a batch. The batch size is arbitrary, and convergence is guaranteed for any size, but higher batch sizes may lead to more accurate gradient estimations and faster convergence. – Matthew Hampsey Oct 11 '16 at 2:57
• That's not what I meant by "multiple inputs at once". I didn't word it very well. I meant you usually forward propagate through several inputs in the training sequence, then back propagate back through them all, then update the weights. So the question is, why propagate through a whole sequence when doing just one input at a time is much easier and still works – Frobot Oct 11 '16 at 3:09
• I think some clarification here is required. When you say "inputs", are you referring to multiple training examples, or are you referring to multiple time steps within a single training example? – Matthew Hampsey Oct 11 '16 at 3:35
• I will post an answer to this question by the end of today. I finished making a BPTT version, just have to train and compare. After that if you still want to see some code let me know what you want to see and I guess I could still post it – Frobot Oct 12 '16 at 15:27

A RNN is a Deep Neural Network (DNN) where each layer may take new input but have the same parameters. BPT is a fancy word for Back Propagation on such a network which itself is a fancy word for Gradient Descent.

Say that the RNN outputs $\hat{y}_t$ in each step and $$error_t=(y_t-\hat{y}_t)^2$$

In order to learn the weights we need gradients for the function to answer the question "how much does a change in parameter effect the loss function?" and move the parameters in the direction given by:

$$\nabla error_t=-2(y_t-\hat{y}_t)\nabla \hat{y}_t$$

I.e we have a DNN where we get feedback on how good the prediction is at each layer. Since a change in parameter will change every layer in the DNN (timestep) and every layer contributes to the forthcoming outputs this needs to be accounted for.

Take a simple one neuron-one layer network to see this semi-explicitly:

\begin{align*} \hat{y}_{t+1} =& f(a+bx_t+c\hat{y}_t)\\ \frac{\partial}{\partial a}\hat{y}_{t+1} = & f'(a+bx_t+c\hat{y}_t)\cdot c\cdot \frac{\partial}{\partial a}\hat{y}_{t} \\ \frac{\partial}{\partial b}\hat{y}_{t+1} = & f'(a+bx_t+c\hat{y}_t)\cdot (x_t+c\cdot\frac{\partial}{\partial b}\hat{y}_{t})\\ \frac{\partial}{\partial c}\hat{y}_{t+1} = & f'(a+bx_t+c\hat{y}_t)\cdot (\hat{y}_t+c\cdot\frac{\partial}{\partial c}\hat{y}_{t})\\ \iff\\ \nabla \hat{y}_{t+1} =& f'(a+bx_t+c\hat{y}_t)\cdot \left(\begin{bmatrix}0\\x_t\\\hat{y}_t \end{bmatrix} + c \mathbin{\color{red}{\nabla \hat{y}_{t}}} \right) \end{align*}

With $\delta$ the learning rate one training step is then: $$\begin{bmatrix}\tilde{a}\\\tilde{b}\\\tilde{c}\end{bmatrix} \leftarrow \begin{bmatrix}a\\b\\c\end{bmatrix} + \delta (y_{t}-\hat{y}_{t})\nabla \hat{y}_t$$

What we see is that in order to calculate $\nabla \hat{y}_{t+1}$ you need to calculate i.e roll out $\nabla \hat{y}_{t}$. What you propose is to simply disregard the red part calculate the red part for $t$ but not recurse further. I assume that your loss is something like

$$error=\sum_t(y_t-\hat{y}_t)^2$$

Maybe each step will then contribute a crude direction which is enough in aggregation? This could explain your results but I'd be really interested in hearing more about your method/loss function! Also would be interested in a comparison with a two timestep windowed ANN.

RNN: Stateful - carry forward hidden state $h_t$ indefinitely This is your model but the training is different.
• If you only forward propagate one step, isn't this a two layered ANN with feature input of last step to the first layer, feature input to the current step at the second layer but has same weights/parameters for both layers? I'd expect similar results or better with an ANN that takes input $\hat{y}_{t+1}=f(x_t,x_{t-1})$ i.e that uses a fixed time-window of size 2. If it only carries forward one step, can it learn long term dependencies? – ragulpr Oct 10 '16 at 20:41
• Look at my updated post. Your architecture is not an RNN, it's stateless so long term-dependencies not explicitly baked into the features can't be learned. Previous predictions does not influence future predictions. You can see this as if $\frac{\partial}{\partial \hat{y}_{t-2}}\hat{y}_t =0$ for your architecture. BPTT is in theory identical to BP but performed on an RNN-architecture so you can't but I see what you mean, and the answer is no. Would be really interesting to see experiments on stateful RNN but only onestep BPTT though ^^ – ragulpr Oct 10 '16 at 22:41