0
$\begingroup$

I am currently reading on RNNs and Backprop through Time. With MLPs using SGD, we did Backprop after every training sample. With RNNs, one method to avoid exploding gradients is to cut an input sample into several samples and do BPTT for each of these "cutted" samples.

I've read about BPTT and RNNs in the Dive into Deep Learning book and my lecture, both of which used text prediction/generation as an example use case. The total loss is the mean loss for all timesteps in a input sequence. For text generation, that makes sense to me because we can validate whether the next predicted word by the model corresponds to the word in out input sequence.

However, I do not understand how this metholodgy applies to other use cases, e.g. sentiment analysis. Consider movie reviews which are of different length. We do not want to predict a next word, instead we want to predict positive or negative. In BPTT, we would for each timestep (input word) calculate the loss. However, as long as the RNN does not change its mind within the sequence, we will just sum up the same output over and over again. Is that how it really works in this use case? I do not understand why we want the output per timestep and not just the finaly output, in this case, on the other hand, we need this time-dependent view in RNNs.

Any help on understanding the loss function in contexts other than next-word prediction is much appreciated. Thank you!

$\endgroup$

1 Answer 1

0
$\begingroup$
  1. It's not necessary that you compute the loss at each timestep for RNNs. You could compute only for the last timestep $L^{(T)}$ and then use BPTT algorithm to get the gradients. However, computing the loss at each timestep won't hurt for the sentiment analysis use case - as it would give you an idea of how the sentiment is varying across the sentence. Assuming the output is binary, you get a sequence of [+ve, -ve, +ve, +ve, +ve, -ve, ...] as output; which greatly helps in understanding the sentence, at the cost of greater complexity.

  2. A comment on your statement ...In BPTT, we would for each timestep (input word) calculate the loss... - that is not entirely true for the case in which you only compute the loss at the last timestep $L^{(T)}$. This lecture from CS224 2019 has a great explanation on BPTT (from 35:48). enter image description here

  3. Let's say we want to compute gradient updates for parameter $W_h$. $$\frac{dL^{(T)}}{dW_h} = \frac{dL^{(T)}}{dh^{(T)}} \frac{dh^{(T)}}{dW_h}$$ You can see that the computation kinda blows up - $W_h$ influences each of $h^{(1)}, h^{(2)}, ..., h^{(T)}$ -- making the gradient computation tricky. The trick that the lecture suggests is to consider $W_h$ at each timestep $i$ as a different entity $W_h |_{(i)}$. From the image; using chain rule, what we get is $$\frac{dL^{(T)}}{dW_h} = \Sigma_{i = 1}^{T} \frac{dL^{(T)}}{dW_h|_{(i)}}$$

  4. Great. Now let's shift focus on the individual entities $\frac{dL^{(T)}}{dW_h|_{(i)}}$. Note that $W_h|_{(i)}$ only influences $h^{(i)}, h^{(i+1)}, ..., h^{(T)}$. The equation involving it looks somewhat like $$h^{(i)} = ReLU(W_x|_{(i)}x^{(i)} + W_h|_{(i)}h^{(i-1)})$$ Through the $i^{th}$ hidden state $h^{(i)}$, the gradient that comes backwards (backprop) is $\frac{dL^{(T)}}{dh^{(i)}}$. So, $$\frac{dL^{(T)}}{dW_h|_{(i)}} = \frac{dL^{(T)}}{dh^{(i)}} \frac{dh^{(i)}}{dW_h|_{(i)}}$$ You know both quantities on RHS (one from backprop, and the other from the ReLU equation above) - hence, you can compute $\frac{dL^{(T)}}{dW_h|_{(i)}}$

  5. The math looks a bit tricky at first glance but once you write it down it all makes sense. Coding BPTT is not tricky though, you can see the iterative nature from previous step. I recommend you to check Andrej Karpathy's minimal implementation of Char-RNN here - easy to follow BPTT code.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.