6
$\begingroup$

RNNs as in: Recurrent Neural Networks

LSTMs as in: Long-Short Term Memory Units

ReLU as in: Rectified Linear Units

Leaky ReLU as in: Modified ReLUs that don't "die" when negative values are inputted.

In practice, machine learning practitioners rarely use vanilla RNNs, citing the vanishing gradient problem (where gradients almost die off after not too many time steps because they are small numbers multiplying by each other a lot, making it basically impossible to train) as the reason. LSTMs are known to solve this problem through their more complex architecture that enable for additive relationships between gradients instead of multiplicative; the latter is the culprit of the vanishing gradient problem.

However, with ANNs, ReLUs are known to solve this problem well, given that their gradient is either "off" (0) or "on" (1). Leaky ReLUs have a small gradient instead of being "off". They solve the problem because the gradients do not saturate.

So, why do we need to have such a complex model like LSTMs when RNNs + ReLU should be able to solve the problem? Is it just that LSTMs perform much better, and we don't have great reasoning as to why?

Improve this question
$\endgroup$
3
  • $\begingroup$ Whoever downvoted—care to explain? Would be much more beneficial to everybody. $\endgroup$
    – MCKapur
    Apr 3, 2017 at 8:18
  • 2
    $\begingroup$ I am not the downvoter but this question is currently automatically flagged as low quality for its length and content. Could you spell out some abbreviations, and use the question to body to give a little bit of background to the problem and what your reasoning is, rather than just restating the question title? (You don't need to write a mini-essay but one advantage of putting a couple of sentences down is that it would be easier for future readers to search and find your answer - with the question only phrased once, they need to get very close to your wording to find it.) $\endgroup$
    – Silverfish
    Apr 3, 2017 at 9:02
  • $\begingroup$ @Silverfish Done $\endgroup$
    – MCKapur
    Apr 3, 2017 at 9:25

1 Answer 1

Reset to default
13
$\begingroup$

I think there's some confusion here. The reason you have vanishing gradients in neural networks (with say, softmax) is wholly different from RNNs. With neural networks, you get vanishing gradients because most initial conditions make your outputs end up on either the far left or far right of your softmax layer, giving it a vanishingly small gradient. In general it's difficult to select proper initial conditions, so people opted to use leaky ReLu's because they don't have the above problems.

Whereas with RNN's, the problem is that you are repeatedly applying your RNN to itself, which tends to cause either exponential blowup or shrinkage. See this paper for example:

On the difficulty of training recurrent neural networks: https://arxiv.org/abs/1211.5063

The suggestions of the above paper are: if the gradient is too large, then clip it to a smaller value. If the gradient is too small, regularize it via a soft constraint to not vanish.

There's a lot of research on LSTMs, and plenty of theories on why LSTMs tend to outperform RNNs. Here's a nice explanation: http://r2rt.com/written-memories-understanding-deriving-and-extending-the-lstm.html#fnref2

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Note that the version of the first paper that you linked to is not the most up to date one. This is the most up to date (and this is actually the only version that appears when you search for the paper's name in google scholar). $\endgroup$
    – Oren Milman
    Sep 29, 2018 at 4:45
  • $\begingroup$ Also, IIUC Hinton's explains in coursera.org/lecture/neural-networks/… (around minute 3) that the cause for vanishing gradients is the same for typical feedforward networks and RNN, while the problem is bigger in RNN, as they tend to have much more layers (when they are unfolded) than typical feedforward networks. $\endgroup$
    – Oren Milman
    Sep 29, 2018 at 4:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.