I am trying to understand the architecture of RNNs. I have found this tutorial which has been very helpful: http://colah.github.io/posts/2015-08-Understanding-LSTMs/

Especially this image: enter image description here

How does this fit into a feed-forward network? Is this image just another node in each layer?

  • $\begingroup$ Or is this what every neuron looks like? $\endgroup$ – Adam12344 Oct 29 '15 at 16:25

A is, in fact, a full layer. The output of the layer is $h_t$, is in fact the neuron output, that can be plugged into a softmax layer (if you want a classification for the time step $t$, for instance) or anything else such as another LSTM layer if you want to go deeper. The input of this layer is what sets it apart from the regular feedforward network: it takes both the input $x_t$ and the full state of the network in the previous time step (both $h_{t-1}$ and the other variables from the LSTM cell).

Note that $h_t$ is a vector. So, if you want to make an analogy with a regular feedforward network with 1 hidden layer, then A could be thought as taking the place of all of these neurons in the hidden layer (plus the extra complexity of the recurring part).

  • $\begingroup$ Do $h_t$, $c_t$ and $x_t$ have all the same length? $\endgroup$ – user_1177868 Apr 7 '19 at 22:01

In your image A is a single hidden layer with a single hidden Neuron. From left to right is the time axis, and at the bottom you receive an input at every time. At the top the network could be further expanded by adding layers.

If you would unfold this network in time, as is visually being shown in your picture (from left to right the time-axis is unfolded) then you would obtain a feedforward network with T (total amount of time steps) hidden layers each containing a single node (neuron) as is drawn in the middle A block.

Hope this answers your question.


I'd like to explain that simple diagram in a relatively complicated context: attention mechanism in the decoder of the seq2seq model.

In the flow diagram bellow, $h_0$ to $h_{k-1}$ are time steps(of the same length as the input number with PADs for blanks). Each time the word is put into the ith (time step) LSTM neural(or kernal cell the same as anyone of the three in your image) it calculates the ith output according to its previous state((i-1)th output) and the ith input $x_i$. I illustrate your issue using this is because all the states of the timestep are saved for the attention mechanism rather than just discarded only to get the last one. It is just one neural and is viewed as a layer(multiple layers can be stacked to form for example a bidirectional encoder in some seq2seq models to extract more abstract information in the higer layers).

It then encodes the sentence(with the L words and each one represented as a vector of the shape: embedding_dimention * 1) into a list of L tensors(each of the shape: num_hidden/num_units * 1). And the state past to the decoder is just the last vector as the sentence embedding of the same shape of each item in the list.

enter image description here
Picture source: Attention Mechanism


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.