I have a question about the mechanism of vanishing gradients in RNN's. There's a related question here (Does the vanishing gradient in RNNs present a problem?) but I want to ask some more specific questions.

I'm taking Andrew Ng's specialization in deep learning on Coursera, specifically the sequence model course. The way he explained the problem is that you can have sentences of unlimited length, such as "The cat, which was gray, and happy, and small, ..., was hungry". Contrast that with a different long sentence like "The cats, which were gray, and happy, and small, ..., were hungry"

The intuition he gave is that when vanishing gradients occur, the RNN can have difficulty distinguishing between "was" and "were" because the noun on which the word depends is much earlier in the sentence. That's as far as his explanation went.

I've had some difficulty wrapping my head around this explanation. I decided to break the problem down and try to explain it to myself. Here is what I came up with... can you tell me if I'm right?

1) There are three sets of weights in an RNN, which are the same at each time step (I made up these terms but it's helpful to the explanation): input weights are applied directly to the input, and context weights are applied to the prior context (the prior activations). These two are applied simultaneously to create what I'll call the main activations at the particular time step. Then the output weights are used to get the final output at that time step.

2) The first two weights (input and context) are interesting because they have to serve two roles: They have to help predict the output at the given timestep, and they also have to help propagate information to the next timestep.

3) Backpropagation would happen once, separately, for each word in the series.

4) The very beginning of backpropagation wouldn't suffer from vanishing gradients, so the weights would always be trained appropriately for role one: translating input/context into a proper prediction at the given timestep.

5) However, the end of backpropagation would suffer from vanishing gradients. Therefore the weights would not be properly trained for role two: packaging information so that future layers can make use of it.

6) Therefore when making predictions on a long sentence like the one in the example, when we get to the word "cats" we would probably get the correct output (e.g. 'chat' if we are translating into French). Or at least, if we get it wrong, it wouldn't be due to vanishing gradients. This is because this prediction mainly relies on the input, and there isn't any interesting context. So role one is enough.

7) However when it gets to the end of the sentence, now it needs to make a prediction that ideally relies not only on the current input, but also on the context. So while the weights at this time step are actually well-trained for role one ("translating input/context into a proper prediction at the given timestep"), they are receiving garbage contextual input, and as they say, "garbage in, garbage out." The prediction may not work well, not because the weights at this time step are bad, but because the weights at a prior time step were not up to the task of role two: packaging information so that future layers can make use of it.

Does that make sense / is it correct?

  • $\begingroup$ I thought you just made it too complicated. The intuition is that we are applying the same matrices again and again for each character in each time step. $\endgroup$ Feb 6, 2020 at 12:30


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.