RNNs as in: Recurrent Neural Networks
LSTMs as in: Long-Short Term Memory Units
ReLU as in: Rectified Linear Units
Leaky ReLU as in: Modified ReLUs that don't "die" when negative values are inputted.
In practice, machine learning practitioners rarely use vanilla RNNs, citing the vanishing gradient problem (where gradients almost die off after not too many time steps because they are small numbers multiplying by each other a lot, making it basically impossible to train) as the reason. LSTMs are known to solve this problem through their more complex architecture that enable for additive relationships between gradients instead of multiplicative; the latter is the culprit of the vanishing gradient problem.
However, with ANNs, ReLUs are known to solve this problem well, given that their gradient is either "off" (0) or "on" (1). Leaky ReLUs have a small gradient instead of being "off". They solve the problem because the gradients do not saturate.
So, why do we need to have such a complex model like LSTMs when RNNs + ReLU should be able to solve the problem? Is it just that LSTMs perform much better, and we don't have great reasoning as to why?