Why back propagate through time in a RNN?

Question

In a recurrent neural network, you would usually forward propagate through several time steps, "unroll" the network, and then back propagate across the sequence of inputs.

Why would you not just update the weights after each individual step in the sequence? (the equivalent of using a truncation length of 1, so there is nothing to unroll) This completely eliminates the vanishing gradient problem, greatly simplifies the algorithm, would probably reduce the chances of getting stuck in local minima, and most importantly seems to work fine. I trained a model this way to generate text and the results seemed comparable to results I have seen from BPTT trained models. I am only confused on this because every tutorial on RNNs I have seen says to use BPTT, almost as if it is required for proper learning, which is not the case.

Update: I added an answer

Answer 1

Edit: I made a big mistake when comparing the two methods and have to change my answer. It turns out the way I was doing it, just back propagating on the current time step, actually starts out learning faster. The quick updates learn the most basic patterns very quickly. But on a larger data set and with longer training time, BPTT does in fact come out on top. I was testing a small sample for just a few epochs and assumed whoever starts out winning the race will be the winner. But this did lead me to an interesting find. If you start out your training back propagating just a single time step, then change to BPTT and slowly increase how far back you propagate, you get faster convergence.

Answer 2

"Unfolding through time" is simply an application of the chain rule, $$\frac{dF(g(x), h(x), m(x))}{dx} = \frac{\partial F}{\partial g}\frac{dg}{dx} + \frac{\partial F}{\partial h}\frac{dh}{dx} + \frac{\partial F}{\partial m}\frac{dm}{dx}$$

The output of an RNN at time step $t$, $H_t$ is a function of the parameters $\theta$, the input $x_t$ and the previous state, $H_{t-1}$ (note that instead $H_t$ may be transformed again at time step $t$ to obtain the output, that is not important here). Remember the goal of gradient descent: given some error function $L$, let's look at our error for the current example (or examples), and then let's adjust $\theta$ in such a way, that given the same example again, our error would be reduced.

How exactly did $\theta$ contribute to our current error? We took a weighted sum with our current input, $x_t$, so we'll need to backpropagate through the input to find $\nabla_\theta a(x_t, \theta)$, to work out how to adjust $\theta$. But our error was also the result of some contribution from $H_{t-1}$, which was also a function of $\theta$, right? So we need to find out $\nabla_\theta H_{t-1}$, which was a function of $x_{t-1}$, $\theta$ and $H_{t-2}$. But $H_{t-2}$ was also a function a function of $\theta$. And so on.

Answer 3

A RNN is a Deep Neural Network (DNN) where each layer may take new input but have the same parameters. BPT is a fancy word for Back Propagation on such a network which itself is a fancy word for Gradient Descent.

Say that the RNN outputs $\hat{y}_t$ in each step and \begin{equation} error_t=(y_t-\hat{y}_t)^2 \end{equation}

In order to learn the weights we need gradients for the function to answer the question "how much does a change in parameter effect the loss function?" and move the parameters in the direction given by:

\begin{equation} \nabla error_t=-2(y_t-\hat{y}_t)\nabla \hat{y}_t \end{equation}

I.e we have a DNN where we get feedback on how good the prediction is at each layer. Since a change in parameter will change every layer in the DNN (timestep) and every layer contributes to the forthcoming outputs this needs to be accounted for.

Take a simple one neuron-one layer network to see this semi-explicitly:

\begin{align*} \hat{y}_{t+1} =& f(a+bx_t+c\hat{y}_t)\\ \frac{\partial}{\partial a}\hat{y}_{t+1} = & f'(a+bx_t+c\hat{y}_t)\cdot c\cdot \frac{\partial}{\partial a}\hat{y}_{t} \\ \frac{\partial}{\partial b}\hat{y}_{t+1} = & f'(a+bx_t+c\hat{y}_t)\cdot (x_t+c\cdot\frac{\partial}{\partial b}\hat{y}_{t})\\ \frac{\partial}{\partial c}\hat{y}_{t+1} = & f'(a+bx_t+c\hat{y}_t)\cdot (\hat{y}_t+c\cdot\frac{\partial}{\partial c}\hat{y}_{t})\\ \iff\\ \nabla \hat{y}_{t+1} =& f'(a+bx_t+c\hat{y}_t)\cdot \left(\begin{bmatrix}0\\x_t\\\hat{y}_t \end{bmatrix} + c \mathbin{\color{red}{\nabla \hat{y}_{t}}} \right) \end{align*}

With $\delta$ the learning rate one training step is then: \begin{equation} \begin{bmatrix}\tilde{a}\\\tilde{b}\\\tilde{c}\end{bmatrix} \leftarrow \begin{bmatrix}a\\b\\c\end{bmatrix} + \delta (y_{t}-\hat{y}_{t})\nabla \hat{y}_t \end{equation}

What we see is that in order to calculate $\nabla \hat{y}_{t+1}$ you need to calculate i.e roll out $\nabla \hat{y}_{t}$. What you propose is to simply disregard the red part calculate the red part for $t$ but not recurse further. I assume that your loss is something like

\begin{equation} error=\sum_t(y_t-\hat{y}_t)^2 \end{equation}

Maybe each step will then contribute a crude direction which is enough in aggregation? This could explain your results but I'd be really interested in hearing more about your method/loss function! Also would be interested in a comparison with a two timestep windowed ANN.

edit4: After reading comments it seems like your architecture is not an RNN.

RNN: Stateful - carry forward hidden state $h_t$ indefinitely This is your model but the training is different.

Your model: Stateless - hidden state rebuilt in each step edit2 : added more refs to DNNs edit3 : fixed gradstep and some notation edit5 : Fixed the interpretation of your model after your answer/clarification.