17
$\begingroup$

In a recurrent neural network, you would usually forward propagate through several time steps, "unroll" the network, and then back propagate across the sequence of inputs.

Why would you not just update the weights after each individual step in the sequence? (the equivalent of using a truncation length of 1, so there is nothing to unroll) This completely eliminates the vanishing gradient problem, greatly simplifies the algorithm, would probably reduce the chances of getting stuck in local minima, and most importantly seems to work fine. I trained a model this way to generate text and the results seemed comparable to results I have seen from BPTT trained models. I am only confused on this because every tutorial on RNNs I have seen says to use BPTT, almost as if it is required for proper learning, which is not the case.

Update: I added an answer

Improve this question
$\endgroup$
4
  • $\begingroup$ An interesting direction to take this research would be to compare the results that you've achieved on your problem with benchmarks published in the literature on standard RNN problems. That would make a really cool article. $\endgroup$
    – Sycorax ♦
    Oct 10, 2016 at 19:40
  • $\begingroup$ Your "Update: I added an answer" replaced the previous edit with your architecture description and an illustration. Is it on purpose? $\endgroup$
    – amoeba
    Oct 12, 2016 at 23:01
  • $\begingroup$ Yes I took it out because it didn't seem relevant to the actual question really and it took up a lot of space, but I can add it back if it helps $\endgroup$
    – Frobot
    Oct 12, 2016 at 23:08
  • $\begingroup$ Well people seem to have massive problems with understanding your architecture, so I guess any additional explanations are useful. You can add it to your answer instead of your question, if you prefer. $\endgroup$
    – amoeba
    Oct 13, 2016 at 16:22

3 Answers 3

Reset to default
6
$\begingroup$

Edit: I made a big mistake when comparing the two methods and have to change my answer. It turns out the way I was doing it, just back propagating on the current time step, actually starts out learning faster. The quick updates learn the most basic patterns very quickly. But on a larger data set and with longer training time, BPTT does in fact come out on top. I was testing a small sample for just a few epochs and assumed whoever starts out winning the race will be the winner. But this did lead me to an interesting find. If you start out your training back propagating just a single time step, then change to BPTT and slowly increase how far back you propagate, you get faster convergence.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Thank you for your update. In the source of that last image he says this about the one to one setting: "Vanilla mode of processing without RNN, from fixed-sized input to fixed-sized output (e.g. image classification)." So that's what we were saying. If it's as you've described it has no state and it's not an RNN. "forward propagating through a single input before back propagating" - I'd call that an ANN. But these wouldn't perform as good with text so something's up and I have no idea what because I don't have the code $\endgroup$
    – ragulpr
    Oct 13, 2016 at 0:18
  • $\begingroup$ I didn't read that part and you are correct. The model I am using is actually the "many to many" on the far right. i assumed in the "one to one" section there were really many of these all connected and the drawing just left it out. but that actually is one of the options on the far right that i didn't notice (it's odd to have that one in there in a blog about RNNs, so i assumed they were all recurrent). I will edit that part of the answer to make more sense $\endgroup$
    – Frobot
    Oct 13, 2016 at 0:32
  • $\begingroup$ I imagined that was the case, that's why I insisted on seeing your loss function. If it's many to many your loss is akin to $error=\sum_t(y_t-\hat{y}_t)^2$ and it's identically an RNN and you're propagating/inputing the whole sequence but then just truncating BPTT i.e you'd calculate the red part in my post but not recurse further. $\endgroup$
    – ragulpr
    Oct 13, 2016 at 0:51
  • $\begingroup$ My loss function doesn't sum over time. I take one input, get one output, then calculate a loss, and update the weights, then move on to t+1, so there is nothing to sum. I will add the exact loss function to the original post $\endgroup$
    – Frobot
    Oct 13, 2016 at 0:58
  • $\begingroup$ Just post your code I'm not doing any more guessing, this is silly. $\endgroup$
    – ragulpr
    Oct 13, 2016 at 1:00
4
$\begingroup$

"Unfolding through time" is simply an application of the chain rule, $$\frac{dF(g(x), h(x), m(x))}{dx} = \frac{\partial F}{\partial g}\frac{dg}{dx} + \frac{\partial F}{\partial h}\frac{dh}{dx} + \frac{\partial F}{\partial m}\frac{dm}{dx}$$

The output of an RNN at time step $t$, $H_t$ is a function of the parameters $\theta$, the input $x_t$ and the previous state, $H_{t-1}$ (note that instead $H_t$ may be transformed again at time step $t$ to obtain the output, that is not important here). Remember the goal of gradient descent: given some error function $L$, let's look at our error for the current example (or examples), and then let's adjust $\theta$ in such a way, that given the same example again, our error would be reduced.

How exactly did $\theta$ contribute to our current error? We took a weighted sum with our current input, $x_t$, so we'll need to backpropagate through the input to find $\nabla_\theta a(x_t, \theta)$, to work out how to adjust $\theta$. But our error was also the result of some contribution from $H_{t-1}$, which was also a function of $\theta$, right? So we need to find out $\nabla_\theta H_{t-1}$, which was a function of $x_{t-1}$, $\theta$ and $H_{t-2}$. But $H_{t-2}$ was also a function a function of $\theta$. And so on.

Improve this answer
$\endgroup$
22
  • $\begingroup$ I understand why you back propagate through time in a traditional RNN. I'm trying to find out why a traditional RNN uses multiple inputs at once for training, when using just one at a time is much simpler and also works $\endgroup$
    – Frobot
    Oct 11, 2016 at 2:46
  • $\begingroup$ The only sense in which you can feed in multiple inputs at once into an RNN is feeding in multiple training examples, as part of a batch. The batch size is arbitrary, and convergence is guaranteed for any size, but higher batch sizes may lead to more accurate gradient estimations and faster convergence. $\endgroup$
    – Matthew Hampsey
    Oct 11, 2016 at 2:57
  • $\begingroup$ That's not what I meant by "multiple inputs at once". I didn't word it very well. I meant you usually forward propagate through several inputs in the training sequence, then back propagate back through them all, then update the weights. So the question is, why propagate through a whole sequence when doing just one input at a time is much easier and still works $\endgroup$
    – Frobot
    Oct 11, 2016 at 3:09
  • $\begingroup$ I think some clarification here is required. When you say "inputs", are you referring to multiple training examples, or are you referring to multiple time steps within a single training example? $\endgroup$
    – Matthew Hampsey
    Oct 11, 2016 at 3:35
  • 1
    $\begingroup$ I will post an answer to this question by the end of today. I finished making a BPTT version, just have to train and compare. After that if you still want to see some code let me know what you want to see and I guess I could still post it $\endgroup$
    – Frobot
    Oct 12, 2016 at 15:27
2
$\begingroup$

A RNN is a Deep Neural Network (DNN) where each layer may take new input but have the same parameters. BPT is a fancy word for Back Propagation on such a network which itself is a fancy word for Gradient Descent.

Say that the RNN outputs $\hat{y}_t$ in each step and \begin{equation} error_t=(y_t-\hat{y}_t)^2 \end{equation}

In order to learn the weights we need gradients for the function to answer the question "how much does a change in parameter effect the loss function?" and move the parameters in the direction given by:

\begin{equation} \nabla error_t=-2(y_t-\hat{y}_t)\nabla \hat{y}_t \end{equation}

I.e we have a DNN where we get feedback on how good the prediction is at each layer. Since a change in parameter will change every layer in the DNN (timestep) and every layer contributes to the forthcoming outputs this needs to be accounted for.

Take a simple one neuron-one layer network to see this semi-explicitly:

\begin{align*} \hat{y}_{t+1} =& f(a+bx_t+c\hat{y}_t)\\ \frac{\partial}{\partial a}\hat{y}_{t+1} = & f'(a+bx_t+c\hat{y}_t)\cdot c\cdot \frac{\partial}{\partial a}\hat{y}_{t} \\ \frac{\partial}{\partial b}\hat{y}_{t+1} = & f'(a+bx_t+c\hat{y}_t)\cdot (x_t+c\cdot\frac{\partial}{\partial b}\hat{y}_{t})\\ \frac{\partial}{\partial c}\hat{y}_{t+1} = & f'(a+bx_t+c\hat{y}_t)\cdot (\hat{y}_t+c\cdot\frac{\partial}{\partial c}\hat{y}_{t})\\ \iff\\ \nabla \hat{y}_{t+1} =& f'(a+bx_t+c\hat{y}_t)\cdot \left(\begin{bmatrix}0\\x_t\\\hat{y}_t \end{bmatrix} + c \mathbin{\color{red}{\nabla \hat{y}_{t}}} \right) \end{align*}

With $\delta$ the learning rate one training step is then: \begin{equation} \begin{bmatrix}\tilde{a}\\\tilde{b}\\\tilde{c}\end{bmatrix} \leftarrow \begin{bmatrix}a\\b\\c\end{bmatrix} + \delta (y_{t}-\hat{y}_{t})\nabla \hat{y}_t \end{equation}

What we see is that in order to calculate $\nabla \hat{y}_{t+1}$ you need to calculate i.e roll out $\nabla \hat{y}_{t}$. What you propose is to simply disregard the red part calculate the red part for $t$ but not recurse further. I assume that your loss is something like

\begin{equation} error=\sum_t(y_t-\hat{y}_t)^2 \end{equation}

Maybe each step will then contribute a crude direction which is enough in aggregation? This could explain your results but I'd be really interested in hearing more about your method/loss function! Also would be interested in a comparison with a two timestep windowed ANN.

edit4: After reading comments it seems like your architecture is not an RNN.

RNN: Stateful - carry forward hidden state $h_t$ indefinitely Statefull This is your model but the training is different.

Your model: Stateless - hidden state rebuilt in each step stateless edit2 : added more refs to DNNs edit3 : fixed gradstep and some notation edit5 : Fixed the interpretation of your model after your answer/clarification.

Improve this answer
$\endgroup$
13
  • 1
    $\begingroup$ thank you for your answer. I think you may have misunderstood what I am doing though. In the forward propagation I only do one step, so that in the back propagation it is also only one step. I don't forward propagate across multiple inputs in the training sequence. I see what you mean about a crude direction that is enough in aggregation to allow learning, but I have checked my gradients with numerically calculated gradients and they match for 10+ decimal places. The back prop works fine. I am using cross entropy loss. $\endgroup$
    – Frobot
    Oct 10, 2016 at 20:18
  • 1
    $\begingroup$ I am working on taking my same model and retraining it with BPTT as we speak to have a clear comparison. I have also trained a model using this "one step" algorithm to predict whether a stock price will rise or fall for the next day, which is getting decent accuracy, so I will have two different models to compare BPTT vs single step back prop. $\endgroup$
    – Frobot
    Oct 10, 2016 at 20:22
  • $\begingroup$ If you only forward propagate one step, isn't this a two layered ANN with feature input of last step to the first layer, feature input to the current step at the second layer but has same weights/parameters for both layers? I'd expect similar results or better with an ANN that takes input $\hat{y}_{t+1}=f(x_t,x_{t-1})$ i.e that uses a fixed time-window of size 2. If it only carries forward one step, can it learn long term dependencies? $\endgroup$
    – ragulpr
    Oct 10, 2016 at 20:41
  • 1
    $\begingroup$ I'm using a sliding window of size 1, but the results are vastly different than making a sliding window of size 2 ANN with inputs (xt,xt−1). I can purposely let it overfit when learning a huge body of text and it can reproduce the entire text with 0 errors, which requires knowing long term dependencies that would be impossible if you only had (xt,xt−1) as input. the only question I have left is if using BPTT would allow the dependencies to become longer, but it honestly doesn't look like it would. $\endgroup$
    – Frobot
    Oct 10, 2016 at 20:48
  • $\begingroup$ Look at my updated post. Your architecture is not an RNN, it's stateless so long term-dependencies not explicitly baked into the features can't be learned. Previous predictions does not influence future predictions. You can see this as if $\frac{\partial}{\partial \hat{y}_{t-2}}\hat{y}_t =0$ for your architecture. BPTT is in theory identical to BP but performed on an RNN-architecture so you can't but I see what you mean, and the answer is no. Would be really interesting to see experiments on stateful RNN but only onestep BPTT though ^^ $\endgroup$
    – ragulpr
    Oct 10, 2016 at 22:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.