1
$\begingroup$

I was watching a video* about making RNNs deeper by adding additional hidden layers. If time is dimension 1, then these hidden layers are dimension 2, so the neural net looks like a matrix - see the screenshot below which makes it clearer.

Prof. Ng mentioned that unlike deep neural nets which can have over 100 hidden layers, it's uncommon to have an RNN with more than, say, 3 hidden layers. This is because the network also expands along the time dimension, so it can already become quite large.

He also mentioned that sometimes instead of directly predicting y-hat after the third RNN hidden layer, people will attach ordinary neural networks to the top, and run say 20 additional hidden layers before making a prediction. These hidden layers are run for every time step but they are NOT connected horizontally. You can see where he drew them above y-hat 1 and y-hat 2, and they also apply to y-hat 3 and 4.

What doesn't make sense to me is: I don't see why this alternative is less computationally expensive. If the primary computational expense is multiplying inputs with weights to get activations, that expense still applies to the "normal" neural networks. 1 additional hidden layer in the "normal" networks creates the same number of matrix multiplications as 1 additional RNN hidden layer: the number is equal to the number of time-steps.

I suppose the RNN part could be slower since parts have to run sequentially, but actually you could partially parallelize that as well (you can calculate both a[2]<1> and a[1]<2> at the same time, for example).

Any ideas?

*in Andrew Ng's course on sequence models on Coursera

enter image description here

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The key idea here is that RNN steps cannot be parallelized because they are connected: each one needs the result of the previous one to run.

If they were not connected, RNN would not be particularly more expensive than other models.

However, if you collect all the results from you RNN (with $N$ hidden layers) at each time step, you can then apply a standard multi-perceptron with $M$ layers to all these outputs in parallel. Thus, the computation of your $M$ layers is much faster than the $N$ ones of the RNN if $N$ and $M$ are "quite close"...

Improve this answer
$\endgroup$
3
  • $\begingroup$ I was thinking about that, but it seems like you could get near-parallel processing in an RNN. Say that each timestep is a thread. Then you could start the thread for the second timestep once the first hidden layer is complete for the first timestep, you could start the thread for the third timestep once the first hidden layer is complete for the second timestep, etc. You'd also need to coordinate your threads so that they retain this ordering as they move down the hidden layers. So it's pretty complex and maybe locking costs computation too, but in theory it could be done. $\endgroup$
    – Stephen
    Commented Apr 13, 2018 at 14:11
  • $\begingroup$ Also I didn't understand the grammar of your point at the very end, "for N and M 'quite close'..." $\endgroup$
    – Stephen
    Commented Apr 13, 2018 at 14:12
  • $\begingroup$ At least for M and N equal or close, a multilayer perceptron should be faster than an RNN. $\endgroup$
    – Pop
    Commented Apr 13, 2018 at 14:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.