I was watching a video* about making RNNs deeper by adding additional hidden layers. If time is dimension 1, then these hidden layers are dimension 2, so the neural net looks like a matrix - see the screenshot below which makes it clearer.
Prof. Ng mentioned that unlike deep neural nets which can have over 100 hidden layers, it's uncommon to have an RNN with more than, say, 3 hidden layers. This is because the network also expands along the time dimension, so it can already become quite large.
He also mentioned that sometimes instead of directly predicting y-hat after the third RNN hidden layer, people will attach ordinary neural networks to the top, and run say 20 additional hidden layers before making a prediction. These hidden layers are run for every time step but they are NOT connected horizontally. You can see where he drew them above y-hat 1 and y-hat 2, and they also apply to y-hat 3 and 4.
What doesn't make sense to me is: I don't see why this alternative is less computationally expensive. If the primary computational expense is multiplying inputs with weights to get activations, that expense still applies to the "normal" neural networks. 1 additional hidden layer in the "normal" networks creates the same number of matrix multiplications as 1 additional RNN hidden layer: the number is equal to the number of time-steps.
I suppose the RNN part could be slower since parts have to run sequentially, but actually you could partially parallelize that as well (you can calculate both
a<2> at the same time, for example).
*in Andrew Ng's course on sequence models on Coursera