1
$\begingroup$

I read online that LSTMs are just RNNs that calculate neuron activations differently. I am not clear on the terminology though. Do we actually call these activations, because they refer not to the output of each neuron but instead to the function that determines the state of each neuron (which the neurons of state-less networks do not have of course).

Is it appropriate to say that LSTMs are simply RNNs with a different activation function? (Instead of something like sigmoid, relu, tanh, etc. LSTMs use... well, I'm not sure if it has a name, but it's a particular series of operations that are distinct from any other activation function).

$\endgroup$

1 Answer 1

1
$\begingroup$

RNN is an abbreviation for Recurrent Neural Networks, a family of models to which models such as LSTM (Long Short Term Memory) and GRU (Gated Recurrent Unit) belong. The main feature of recurrent networks is taking as input an output produced by itself at previous step. What follows, RNNs share weights used for different data points and can take as inputs sequences of any length. There are vanilla RNNs that do just that, below you can find a figure illustrating it.

recurrent neural network

In comparison, LSTM has additional components, the “forget gate” and “input gate”, shown on the diagram below. The role of those gates is to cancel and amplify signals, instead of just passing them forward. LSTMs are one of the solutions for vanishing gradient problem observed in RNNs.

LSTM

Both images are taken from this blog, that goes into more details.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.