Why this Way of iteration in RNNs? One of the things that make them super-slow in training, is that they often use two nested loops to iterate on data through all time steps for every single iteration, calculating loss and cost at the end, and then back-propagate that error to get the gradients to update all weight matrices.
I mean, We can't get rid of iterating for some epochs to fit the model more and more.
but Isn't there any known method to just feeding the whole sequence at once and iterate for some epochs while training?  I know this is impossible if we are talking about something like language models where the next input character is based on the probability made by the softmax activation from the previous time-step.
 A: If I understand correctly, the question you are asking is, why bother with considering batches instead of passing the whole dataset at once. 
There are 2 reasons
1) If you pass the whole dataset, each iteration is much slower since each matrix multiply costs a lot more. For very large datasets (and limited RAM) it may not even be possible to pass the entire dataset. To visualize this, think of passing the information in all of wikipedia through your solver in one iteration.
2) Turns out that the batch method surprisingly works better/iteration than the naive implementation. This fact is mostly empirical. 
Details of the various approaches and reasons for them can be found inhttp://cs231n.github.io/optimization-1/
============================
In some cases, the  entire sentence can be passed all at once. An example of this is the use of CNN for sentence evaluation. The architecture of the word stacking is shown in figure 1 of https://arxiv.org/pdf/1408.5882.pdf. In translation/other sequence models, it becomes painful to send the entire sequence; one reason is the size of the model, and the other is the difficulty in describing a loss function. 
I guess the answer is very application dependent. If you pass an entire sequence, a "correct result" in a translation model I am guessing would be the entire sequence translated properly. Here, you will notice that getting the answer "correct" is much harder. In the sequence approach, if you pass in the data word by word, you can have a notion of partial wins; some percentage of the words can be classified accurately, and you will still get an improvement in accuracy. The notion of a correct answer in translation is very difficult because different translators will have different notions of "correctness" (hence metrics like BLEU). Thus a sentence which gets a label of 1 in one translation can get a label of 0 in another if you pass the entire sequence. The problem is accentuated if you pass the entire document
A: I'm not aware of a good algorithm that takes the whole sequence of an RNN at once when the sequence is very long. Training on partial sequences, doing backprop, and then training on the next partial sequence is called truncated back-propagation through time (TBTT). 
The reason that TBTT is used is that a very long sequence has two problems. 


*

*A very long sequence has lots of computation required to compute both the forward and the backward pass and

*The gradient vanishes as you go farther back in time, so there's no real "gain" to go back very far in time.
Some resources on this include
Williams and Zipser (1992), Gradient-Based Learning Algorithms for Recurrent Networks and Their Computation Complexity and Williams and Peng (1990), An Efficien Gradient-Based Algorithm for On-Line Training of Recurrent Network Trajectories and Ilya Sutskever’s Ph.D. thesis.
