2
$\begingroup$

So I am reading up on how RNN's are different from feed forward neural networks. I understand that it uses the time-1 during training. However it is not clear to me at what "level" this happens.

Does a RNN during the training of a sentence like "I am going out." Pass every word individually through the entire network before it evaluates the prediction or am I picturing this wrong? So does it:

  1. Pass "I" through all layers of the network.
  2. Save that output state as t
  3. Pass "am" and t through all layers of the network.
  4. Save that output state as t
  5. Pass "going" and t through all layers of the network.
  6. Save that output state as t
  7. Pass "out" and t through all layers of the network.
  8. Save that output state as t
  9. Pass "." and t through all layers of the network.
  10. Make a prediction
  11. Evaluate prediction
  12. Back propagation
$\endgroup$

1 Answer 1

1
$\begingroup$

Yes, your idea is right. When unrolled in time, the RNN looks like this

  I    am  going  out   .
  ↓     ↓    ↓     ↓    ↓
  █  →  █  → █  →  █  →  █ →  □

where █ is the recurrently applied function and □ is the classifier. The so-called back-propagation in time is nothing but pretending that the RNN is a very deep feed-forward network with the same layer repeated many times. Because the parameters in al the █ layers are the same, the gradient gets summed up.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.