1
$\begingroup$

I have few doubts related to training Elman RNN using Backpropagation Through Time Algorithm.

Assume, I present a sequence to the network and the network adapts the parameters based on the error gradient including hidden state input(context units). Now, when I present the next sequence what should be the starting hidden state input(context units)? Should the hidden state input(context units) be the ones updated by the gradient decent one or the hidden state activation obtained by last input of the previous sequence?

PS: This is a follow-up question How to train Elman RNN for Temporal XOR?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Since temporal context is only valid within the presentation of a specific sequence, the processing of a new sequence should have the context units reset to 0. Otherwise you'd consider the last input of the previous sequence as context which is conceptually wrong (multiple sequences aren't presented in a specific order, if they would, they would be one sequence).

From there, you do exactly what you did with the previous sequence, i.e. you feed the values forward to the hidden layer and from there fill the context layer which is used by the 2nd input of the 2nd sequence.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for clearing my doubt. One last thing, what's the usual practice, does one train the initial context units (by setting it to random value and adjusting the values) or set the initial context units to zero and do not worry about adjusting it's initial value? $\endgroup$
    – Rudra Murthy
    May 29, 2015 at 8:56
  • $\begingroup$ Set it to 0, so it doesn't affect the 1st iteration of backpropagation in the sequence. Setting it to random values would give you a random/wrong context and thus distorted input to the activation function at the hidden layer. Edit: It's like normal Feedforward NNs: Randomization is only used for feedforward weights. $\endgroup$
    – runDOSrun
    May 29, 2015 at 9:06
  • $\begingroup$ What about updating the weights for the initial context layer? If I'm e.g. taking the average weight updates by dividing by k, I should still divide by k to get these weight updates (as opposed to e.g. k-1, as the input context layer are all 0's)? $\endgroup$
    – Tahlor
    Feb 23, 2018 at 18:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.