I have few doubts related to training Elman RNN using Backpropagation Through Time Algorithm.

Assume, I present a sequence to the network and the network adapts the parameters based on the error gradient including hidden state input(context units). Now, when I present the next sequence what should be the starting hidden state input(context units)? Should the hidden state input(context units) be the ones updated by the gradient decent one or the hidden state activation obtained by last input of the previous sequence?

PS: This is a follow-up question How to train Elman RNN for Temporal XOR?

up vote 1 down vote accepted

Since temporal context is only valid within the presentation of a specific sequence, the processing of a new sequence should have the context units reset to 0. Otherwise you'd consider the last input of the previous sequence as context which is conceptually wrong (multiple sequences aren't presented in a specific order, if they would, they would be one sequence).

From there, you do exactly what you did with the previous sequence, i.e. you feed the values forward to the hidden layer and from there fill the context layer which is used by the 2nd input of the 2nd sequence.

  • Thanks for clearing my doubt. One last thing, what's the usual practice, does one train the initial context units (by setting it to random value and adjusting the values) or set the initial context units to zero and do not worry about adjusting it's initial value? – Rudra Murthy May 29 '15 at 8:56
  • Set it to 0, so it doesn't affect the 1st iteration of backpropagation in the sequence. Setting it to random values would give you a random/wrong context and thus distorted input to the activation function at the hidden layer. Edit: It's like normal Feedforward NNs: Randomization is only used for feedforward weights. – runDOSrun May 29 '15 at 9:06
  • What about updating the weights for the initial context layer? If I'm e.g. taking the average weight updates by dividing by k, I should still divide by k to get these weight updates (as opposed to e.g. k-1, as the input context layer are all 0's)? – Tahlor Feb 23 at 18:32

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.