I always see that people teach the unfolded version of the RNN. enter image description here Do they do that just for the sake of teaching because the unfolded version is simple to explain sequence learning on it?
I read in an article before that it's hard to train the folded version of RNN. Why is that?
Do deep learning packages implement the folded or the unfolded version of RNN?


In short: there is no such thing as folded versus unfolded RNN. There is just RNN.

A graphical diagram of the RNN is simply an informal description of how an RNN model works. The folded and unfolded version of the diagram are just two descriptions of the same model. The folded version is a more compact description, whereas the unfolded version makes it more clear that the hidden state takes on multiple values over time. But they are describing the same thing.

However there is a very similar question which you might be asking, which is "must memory be allocated for the hidden state at each time step, or can i simply overwrite the hidden state at the last time step with the hidden state of the current time."

The answer to this is: training the model via backpropagation requires access to the value of the hidden state at every time step, so all previous hidden states must be stored in memory. This is just a fact of backpropagation. On the other hand, during inference time, it's no longer necessary and the hidden state can overwrite itself, saving memory.

  • $\begingroup$ thank you for your answer. the reason why I asked is because I read this article rolling-and-unrolling-rnns, It says that unfolding the RNN made the back-propagation easier. (you can just read the last 3 paragraphs). And also in here RNN unrolling it says "the backward pass requires the conceptualization of unfolding the network" $\endgroup$ – floyd Jul 26 at 1:53
  • $\begingroup$ @floyd i see. maybe it's just me being a bit pedantic, but i like to separate the implementation of RNNs from how they're drawn, to avoid having to "conceptualize" anything. $\endgroup$ – shimao Jul 26 at 23:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.