Here is the picture of RNN enter image description here

It's said, that hidden layer is fully connected(dash dots). But I don't understand why? For example I don't understand why the one from $a_3$ to $a_1$ exists. I thought information transfers from hidden state $a_{t-1}$ to $a_{t}$ and not other way around. Maybe I am confused by terminology.

Can you explain me?


This is indeed an unconventional way to present the structure of an RNN, and I think your confusion is coming from conflating the time indices with the unit indices. Here's the "unrolled" network, with time indices in superscript:

enter image description here

In this diagram, each arrow represents a full set of connections from one layer to the other. For example, an arrow between two blue blocks actually stands in for 9 arrows between all the recurrent units of one layer to all the units of the next.

| cite | improve this answer | |
  • $\begingroup$ thank you for your comment! So it means, for example, that units in layer 1 are fully connected with units in layer 2? $\endgroup$ – Daniel Yefimov Oct 14 at 18:23
  • 1
    $\begingroup$ If by "layer" you're referring to the time slices, then yes, the units at time [1] are fully connected to the units at time [2]. This corresponds to the dashed arrows in the figure in the original question. $\endgroup$ – tddevlin Oct 14 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.