Are LSTMs just RNNs that use a different activation function?

Question

I read online that LSTMs are just RNNs that calculate neuron activations differently. I am not clear on the terminology though. Do we actually call these activations, because they refer not to the output of each neuron but instead to the function that determines the state of each neuron (which the neurons of state-less networks do not have of course).

Is it appropriate to say that LSTMs are simply RNNs with a different activation function? (Instead of something like sigmoid, relu, tanh, etc. LSTMs use... well, I'm not sure if it has a name, but it's a particular series of operations that are distinct from any other activation function).

Answer 1

RNN is an abbreviation for Recurrent Neural Networks, a family of models to which models such as LSTM (Long Short Term Memory) and GRU (Gated Recurrent Unit) belong. The main feature of recurrent networks is taking as input an output produced by itself at previous step. What follows, RNNs share weights used for different data points and can take as inputs sequences of any length. There are vanilla RNNs that do just that, below you can find a figure illustrating it.

In comparison, LSTM has additional components, the “forget gate” and “input gate”, shown on the diagram below. The role of those gates is to cancel and amplify signals, instead of just passing them forward. LSTMs are one of the solutions for vanishing gradient problem observed in RNNs.

Both images are taken from this blog, that goes into more details.